Source-linked AI summary
StoryBench: A Dynamic Benchmark for Evaluating Long-Term Memory with Multi Turns
Luanbo Wan, Weizhi Ma
TL;DR
Long-term memory is important for LLMs in evolving environments, but existing benchmarks face challenges assessing retention, sequential reasoning, and flexibility. StoryBench evaluates these abilities through branching interactive-fiction narratives and two feedback modes. Experiments find that models struggle to recover from earlier mistakes, while the authors conclude the benchmark reveals significant performance gaps.
Problem
Existing benchmarks face challenges in evaluating LLM knowledge retention, sequential reasoning across dynamic interactions, and flexibility, despite the importance of LTM in evolving environments.
Source: §1 · pp. 1–2Method
StoryBench evaluates LLMs in branching multi-turn interactive-fiction narratives using Immediate Feedback and Self Recovery modes.
Source: §1 · pp. 2–3Results
Models show significant performance gaps on StoryBench; GPT-4o and Claude 3.5 Sonnet are relatively stronger, but all evaluated models struggle with self-recovery.
Source: §1 · p. 2, §7 · p. 13Takeaways & Limitations
The authors conclude that StoryBench provides an effective evaluation of LTM capabilities and highlights the importance of developing more robust memory mechanisms.
Source: §7 · p. 13Its text-based, single interactive-fiction domain and six-chapter dataset may limit generalizability and coverage of longer dependencies.
Source: §6 · p. 13
Abstract
from arXiv · showhide
Long-term memory (LTM) is essential for large language models (LLMs) to achieve autonomous intelligence in complex, evolving environments. Despite increasing efforts in memory-augmented and retrieval-based architectures, there remains a lack of standardized benchmarks to systematically evaluate LLMs' long-term memory abilities. Existing benchmarks still face challenges in evaluating knowledge retention and dynamic sequential reasoning, and in their own flexibility, all of which limit their effectiveness in assessing models' LTM capabilities. To address these gaps, we propose a novel benchmark framework based on interactive fiction games, featuring dynamically branching storylines with complex reasoning structures. These structures simulate real-world scenarios by requiring LLMs to navigate hierarchical decision trees, where each choice triggers cascading dependencies across multi-turn interactions. Our benchmark emphasizes two distinct settings to test reasoning complexity: one with immediate feedback upon incorrect decisions, and the other requiring models to independently trace back and revise earlier choices after failure. As part of this benchmark, we also construct a new dataset designed to test LLMs' LTM within narrative-driven environments. We further validate the effectiveness of our approach through detailed experiments. Experimental results demonstrate the benchmark's ability to robustly and reliably assess LTM in LLMs.
1 Introduction
StoryBench addresses challenges in evaluating long-term knowledge retention, sequential reasoning, and flexibility with a dynamic, branching interactive-fiction benchmark. Its evaluations find that models struggle to recover from earlier mistakes.
1 Introduction
- Existing benchmarks face challenges in evaluating knowledge retention, event-sequence reasoning, and flexibility, despite the importance of LTM for changing environments.The paper distinguishes retention across extended texts from reasoning about latent state changes, causal dependencies, and goal shifts across interactions.Source: §1 · pp. 1–2
- StoryBench uses branching multi-turn narratives with Immediate Feedback and Self Recovery modes to assess knowledge retention and sequential reasoning.Self Recovery continues toward a failure ending without a hint, requiring models to identify and revise earlier decisions themselves.Source: §1 · pp. 2–3
- The benchmark comparison considers long-context retention and continuity, non-linear complexity and dynamics, multi-turn interaction, multi-solution flexibility, and combined LTM and STM use.The paper defines continuity as maintaining coherent understanding of entities, events, and relationships across interactions.Source: §1 · p. 2
- All evaluated models struggle with self-recovery, while GPT-4o and Claude 3.5 Sonnet show relatively stronger long-term retention and sequential reasoning.The evaluation covers four LLMs and more than 80 branching story paths; the authors say the findings reveal distinct memory bottlenecks.Source: §1 · p. 2
- The annotated interactive-fiction dataset combines cohesive continuity, dynamic branching, interdependent events, and multiple solution mechanisms to emulate memory challenges.The benchmark framework is also described as adaptable across scenarios.Source: §1 · p. 3
- Repeated trials are used to strengthen statistical robustness and support performance comparisons.Source: §1 · p. 3
2 Related Work
Prior work develops architectural and explicit-memory strategies for handling long contexts and retaining information. Existing evaluation benchmarks span varied context lengths and tasks, but the paper identifies continuing challenges in assessing retention, sequential reasoning, and flexibility.
2.1 Strategies and Techniques for Enhancing Long-Term Memory
- Long-term-memory strategies range from sparse or recurrent attention and alternative architectures to external retrieval, memory modules, hybrid systems, and context compression.The section also describes efficiency techniques and parameter-efficient fine-tuning as approaches to improve handling or retention of information.Source: §2.1 · p. 3
2.2 Benchmarks for Evaluating Long-Term Memory
- Evaluation benchmarks cover long contexts, varied tasks, synthetic retrieval, reasoning, discourse, and agent-based interactions, yet challenges remain in evaluating retention, sequential reasoning, and flexibility.The paper notes that few agent-based benchmarks explicitly isolate long-memory capabilities, identifying the LTM benchmark for multi-turn conversations as a notable exception.Source: §2.2 · pp. 3–4
3 StoryBench
StoryBench evaluates long-term memory in evolving interactive-fiction narratives, where choices shape later outcomes and require sustained sequential reasoning. Its two task modes and targeted metrics assess memory use across immediate correction and delayed recovery.
3.1 Motivation and Overview
- StoryBench addresses static benchmarks’ limited simulation of dynamic decision-making by embedding memory evaluation in evolving, multi-turn narratives.The narratives require models to integrate short- and long-term context and track relationships and contradictions across interactions.Source: §3.1 · p. 4
3.2 Dynamic Narrative and Multi-Turn Decision-Making
- Narratives link decisions across turns: past actions shape future outcomes, and choices can alter conditions or trigger failure endings.The benchmark also includes multiple acceptable paths at many decision points.Source: §3.2 · p. 4
3.3 Two Task Modes for Evaluating LTM
- Immediate Feedback exposes each wrong choice for a retry, while Self Recovery withholds feedback and requires tracing a later failure back to its origin.The paired modes examine short-term adjustment and long-term causal reasoning under uncertainty.Source: §3.3 · p. 5, Figure 1 · p. 5, Figure 2 · p. 5
3.4 Tailored Metrics for Assessing LTM Models
- StoryBench’s targeted metrics cover knowledge retention and sequential reasoning to assess long-term memory in language models.The paper frames these dimensions as complementary aspects of memory performance.Source: §3.4 · p. 5
3.4.1 Metrics for Knowledge Retention
- Knowledge-retention metrics include average correctness across decisions, first-try accuracy, and the longest consecutive run of correct decisions.The longest-run measure reflects contextual consistency over extended intervals, though the paper calls it less critical than the other metrics.Source: §3.4.1 · p. 6
3.4.2 Metrics for Sequential Reasoning
- Sequential-reasoning metrics separate easy from hard decisions, count retries and repeated errors, and track runtime cost and token consumption.Hard decisions require distant-context recall, latent-state tracking, or multi-step reasoning; runtime and token use are auxiliary efficiency measures rather than direct memory-accuracy measures.Source: §3.4.2 · p. 6
4 Dataset Construction
StoryBench’s dataset uses a branching interactive-fiction narrative to test long-term dependencies, interdependent decisions, and adaptable paths to success. Its graph structure and manually annotated game content preserve branching logic and causal relationships for evaluation.
4.1 Overview
- The story spans multiple turns, with choices that alter later paths; long-range dependencies and interdependent decisions test memory and reasoning, while multiple successful paths reward adaptability.Paths can lead to success or failure endings.Source: §4.1 · p. 7
4.2 Structural Organization
- A directed acyclic graph links scene nodes, which contain narrative fragments, to choice nodes, which define branching decisions, enabling non-linear progression and tracing of causal dependencies.Figure 3 illustrates representative structures, including linear chains, long-term dependencies, interdependent decisions, and multi-solution branches.Source: §4.2 · pp. 7–8, Figure 3 · p. 7
4.3 Data Source and Annotation Process
- The dataset follows The Invisible Guardian from its prologue through Chapter 5, with 311 scene nodes and 86 choice nodes in structured JSON.Manual annotation preserves branching logic, causal relationships, chronological order, and memory checkpoints.Source: §4.1 · p. 7, §4.3 · p. 8
- Scene nodes capture locations, characters, dialogue, and possible ending flags; choice nodes record decision contexts and branching options with distinct identifiers and text.Entries are transcribed from the game and annotated as JSON objects.Source: §4.3 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8
5 Experiments & Results
Across the evaluated models, strong local accuracy did not guarantee successful completion of long decision paths, and self-recovery without feedback was more difficult. The analyses identify weaknesses in contextual consistency and in revising earlier choices across multi-step dependencies.
5.1 Experimental Setup
- Four models were evaluated across two modes; each received 10 trials per mode, with prompting and input handling adjusted to address repeated errors and service constraints.The initial Self Recovery phase used five trials without token limits; a second five-trial phase applied vocabulary filtering and a 5,000-token-per-turn limit.Source: §5.1 · p. 9
5.2 Main Results of Long-Term Memory Performance
- Table 2 reports performance by model in Immediate Feedback mode; the supplied passages do not give its cell values.Source: Table 2 · p. 9
5.2.1 Model Analysis
- Claude 3.5 Sonnet achieved the highest Success Count, while Doubao1.5-pro led knowledge-related accuracy metrics but had significantly lower task completion.The authors argue that completing the story chain is essential to evaluating long-term memory, beyond local accuracy alone.Source: §5.2.1 · p. 10
- Most models showed large Easy–Hard Accuracy gaps; Claude and GPT-4o were more consistent across difficulty levels, while Deepseek-R1 dropped on harder decisions.GPT-4o and Doubao1.5-pro had lower Runtime Cost and Token Consumption than Claude 3.5 and Deepseek-R1.Source: §5.2.1 · p. 10, Figure 7 · p. 11
5.2.2 Insights of Distinctions Between Two Modes
- All models performed worse in Self Recovery, but Claude 3.5 and GPT-4o never reached the nine-error intervention threshold and completed tasks without artificial hints.First-Try Accuracy and Longest Consecutive Correct Sequence increased for several models despite the overall decline.Source: §5.2.2 · p. 11, Figure 8 · p. 12
- Deepseek-R1 was comparatively consistent across modes, which the authors interpret as evidence of accurate revisions during backtracking.Source: §5.2.2 · p. 12
5.3 Failure Case Study
- Models often contradicted earlier narrative facts or character motivations, even when relevant facts were in the prompt; most also backtracked only one or two steps when repairing multi-error decisions.The authors retained these failures to reflect the upper-bound difficulty of long-term memory reasoning.Source: §5.3 · p. 12
- Format mismatches, content-filter blocks, server timeouts, and rare hallucinated explanations occurred but were comparatively infrequent and were retained for completeness.Source: §5.3 · p. 12
6 Limitations
StoryBench’s scope is constrained by its text-based interactive-fiction domain, short narrative length, limited model coverage, and scripted feedback. These constraints may limit generalizability and the breadth of behaviors the benchmark captures.
6 Limitations
- Text-based scenarios from a single interactive-fiction domain may limit generalizability to multimodal, knowledge-intensive, or task-oriented contexts.The authors note that these contexts may require multimodal support.Source: §6 · pp. 12–13
- The dataset’s six chapters and limited context length may not capture long-term dependencies and complex reasoning in more extensive narratives.The authors suggest adding subsequent chapters in future work.Source: §6 · p. 13
- Evaluation covers a limited number of mainstream models and uses scripted self-recovery that cannot capture all forms of natural feedback.The authors attribute limited model coverage to API constraints and cost.Source: §6 · p. 13
7 Conclusions
StoryBench evaluates long-term memory in dynamic, multi-turn narrative environments, and experiments find significant performance gaps in current models. The authors conclude that stronger memory mechanisms are important for future context-aware language agents.
7 Conclusions
- Experiments show significant performance gaps among current models on StoryBench, supporting its difficulty and effectiveness for evaluating long-term memory.The evaluation covers knowledge retention and sequential reasoning in representative mainstream models.Source: §7 · p. 13
- StoryBench assesses long-term memory through complex, dynamic, multi-turn narratives involving story understanding, sequential inference, and flexible correction.Source: §7 · p. 13
- The findings underscore the importance of more robust memory mechanisms and lay groundwork for research toward memory-augmented, context-aware language agents.Source: §7 · p. 13