Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs

1 INTRODUCTION

• Protracted, multi-session conversations require models to maintain consistent memory across lengthy interactions, motivating evaluation of long-history reasoning.The introduction notes that LLMs are used in diverse applications and that some have input windows of up to 1M tokens.Source: §1 · p. 1
• Existing benchmarks often concatenate unrelated short sessions, focus on narrow domains, and emphasize recall over abilities such as contradiction resolution and instruction following.Abrupt topic shifts and separable segments can reduce the need for genuine long-range reasoning.Source: §1 · p. 1
• BEAM contains 100 diverse conversations of 100 K to 10 M tokens and 2,000 human-validated probing questions targeting ten memory dimensions.The generation framework builds conversations from recursively decomposed plans, then creates questions emphasizing complicated and multi-hop reasoning.Source: §1 · p. 2
• LIGHT combines long-term episodic retrieval, recent-turn working memory, and a scratchpad that records salient facts after each turn.At inference, the model uses retrieved episodic content, working memory, and the accumulated scratchpad together to answer questions.Source: §1 · p. 2
• 3.5%–12.69% average improvement over the best-performing baseline, depending on backbone model and conversation length, is reported for LIGHT.Experiments also show that even models with long context windows perform substantially worse as conversations lengthen.Source: §1 · p. 2

2.1 PROBLEM FORMULATION

• The benchmark evaluates conversational systems on predefined memory abilities using ability-specific probes, ground-truth answers, and scoring functions.Each probe is appended to a conversation, and the system response is scored for the ability being tested.Source: §2.1 · p. 3

2.2 BENCHMARK CREATION

• BEAM assesses ten complementary memory abilities, including three introduced here: Instruction Following, Event Ordering, and Contradiction Resolution.Together, the abilities target systems’ capacity to maintain, update, and manipulate information across extended conversations.Source: §2.2 · p. 3

2.2.1 CONVERSATION PLAN GENERATION

• Conversation plans provide coherent, chronological storylines with user profiles, relationships, narratives, and timelines.For 10M-token conversations, the authors create ten interlocking plans using sequential expansion or hierarchical decomposition, then augment plans with targeted points for contradiction resolution, information update, and instruction following.Source: §2.2.1 · pp. 3–4

2.2.2 USER UTTERANCE GENERATION

• User utterances are generated in batches from plan bullet points and prior context to maintain continuity while controlling repetition and dialogue length.The LLaMA-3.3 70B model generates questions, with batch and question counts set by domain and target conversation length.Source: §2.2.2 · p. 4, Table 6 · p. 5

2.2.3 ASSISTANT UTTERANCE GENERATION

• Assistant and user role-playing, with question-detection and follow-up modules, creates bidirectional dialogue with contextual replies and clarifications.Each type of exchange is capped at two rounds to support realism while preventing unbounded loops.Source: §2.2.3 · p. 5

2.3 PROBING QUESTIONS GENERATION

• Probe generation uses plan-linked dialogue evidence to create ability-specific candidate questions, which human evaluators validate.For 10M-token dialogues, candidates are generated across sliding windows over the interlocking plans to preserve topical locality and scalability.Source: §2.3 · p. 5

2.4 EVALUATION

• Validated probes are scored with nugget-based criteria for nine abilities, while Event Ordering uses Kendall tau-b to assess recall and sequence.Responses receive nugget scores of 0, 0.5, or 1; an equivalence detector aligns reported events with nuggets before calculating Kendall tau-b.Source: §2.4 · p. 6

3 LIGHT: IMPROVING MEMORY CAPABILITIES OF LLMS

• LIGHT combines episodic recall, recent-turn working memory, and a scratchpad of salient notes to condition the answer generator alongside the question.The framework selects relevant conversation segments, the most recent dialogue pairs, and scratchpad notes filtered for the question.Source: §3 · pp. 6–7

3.1 RETRIEVAL FROM THE CONVERSATION

• Qwen2.5-32B-AWQ extracts key–value pairs and summaries after each turn; embedded indices retrieve the original dialogue segments corresponding to the question’s top k nearest neighbors.The BAAI/bge-small-en-v1.5 embedding model indexes the pairs and summaries, while the original dialogue segments are retained as values for grounding.Source: §3.1 · p. 7

3.2 SCRATCHPAD FORMATION AND UTILIZATION

• For each dialogue pair, Qwen2.5-32B-AWQ extracts salient content spanning semantic, autobiographical, prospective, and contextual information, which is iteratively merged into the scratchpad.When the scratchpad exceeds 30K tokens, GPT-4.1-nano compresses it into a 15K-token summary; unlike episodic memory, it is supplied directly at inference rather than stored in a retrieval database.Source: §3.2 · p. 7
• At inference, semantic chunking divides the scratchpad, Qwen2.5-32B-AWQ labels chunks relevant or irrelevant to the question, and only relevant chunks reach the response generator.The relevance labels are binary yes/no judgments.Source: §3.2 · p. 7

4.1 EXPERIMENTAL SETUP

• Long-context baselines received full histories, while RAG retrieved the five most similar user–assistant turn-pair documents; at 10M, models used the largest recent segment fitting their context window.Only Llama-4-Scout supports a 10M-token window, but it was excluded because of extreme computational requirements.Source: §4.1 · p. 7

4.2 EMPIRICAL FINDINGS

• +49.1% for Llama-4-Maverick and +44.3% for GPT-4.1-nano at 100K, rising to +155.7% for Llama-4-Maverick at 10M over long-context baselines.At 1M, gains reach +75.9% for GPT-4.1-nano and +60.1% for Qwen2.5-32B. At 10M, LIGHT surpasses Gemini-2.0-flash’s long-context baseline by +57.3% but slightly trails RAG; no baseline natively supports the full 10M context.Source: Table 1 · p. 8, §4.2 · p. 8
• Across ten memory abilities, the largest relative gains were summarization (+160.6%), preference following (+76.5%), and multi-hop reasoning (+27.2%), while contradiction resolution remained weakest.Other reported gains were information extraction (+56.7%), instruction following (+39.5%), and temporal reasoning (+56.3%); all methods performed strongest in abstention.Source: §4.2 · pp. 8–9
• At 10M, removing retrieval, scratchpad, working memory, or noise filtering reduced performance by 8.5%, 3.7%, 5.7%, or 8.3%, respectively.At 100K, removing retrieval did not change performance, while removing scratchpad or noise filtering reduced it by 1.1% or 2.2%; working memory was slightly harmful at 100K and 1M. By 10M, every component was beneficial, and the full architecture performed best.Source: §4.2 · p. 9, Figure 3 · p. 9
• Generated conversations received average human ratings of 4.53 for coherence and flow, 4.57 for realism, and 4.64 for complexity and depth on 5-point scales.The paper also reports a qualitative analysis of LIGHT’s failure modes in Appendix G.Source: §4.2 · p. 10

5 RELATED WORK

• Context windows expanded from 512–2K tokens to 128K–1M and even 10M, with conversational histories highlighted as an important application.The cited models include GPT-2/3 (Radford et al., 2019; Brown et al., 2020), Claude-3, GPT-4-Turbo, Gemini 2.0 (DeepMind, 2025; Anthropic, 2025; OpenAI, 2025a), and Llama 4; drivers include efficient attention, positional encodings, long-context training, and inference optimizations (Beltagy et al., 2020; Wang et al., 2020; Dao et al., 2022; Dai et al., 2019; Peng et al., 2023b; Xiong et al., 2023; Ding et al., 2024; Kwon et al., 2023; Li et al., 2024; Liu et al., 2023).Source: §5 · p. 10
• Persistent-memory approaches complement larger context windows through recurrence, state-space architectures, external memory, summarization, and retrieval-augmented generation.Examples include Transformer-XL and Compressive Transformer (Dai et al., 2019; Rae et al., 2019), RWKV, Mamba, and Hyena (Peng et al., 2023a; Gu & Dao, 2023; Poli et al., 2023), Memformer, RETRO, and RMT (Wu et al., 2020; Borgeaud et al., 2022; Fan et al., 2024), AutoCompressor (Chevalier et al., 2023), and REALM, RAG, and HippoRAG (Guu et al., 2020; Lewis et al., 2020; Jimenez Gutierrez et al., 2024).Source: §5 · p. 10
• Existing conversational-memory benchmarks often use narrow domains, shallow dependencies, or concatenated sessions, while MemoryCode uses template-seeded dialogues in one domain (Rakotonirina et al., 2025).These benchmarks assess recall, temporal reasoning, and multi-session reasoning; this work instead generates coherent single-user conversations and adds contradiction resolution, event ordering, and instruction following.Source: §5 · p. 10

• LIGHT boosts memory performance by an average of 3.5%-12.69%, while standard LLM performance degrades over long contexts.The conclusion attributes the improvement to LIGHT’s memory enhancement framework.Source: §6 · p. 10
• BEAM is built with a scalable framework to generate long, coherent dialogues of up to 10M tokens and diverse memory probes.Source: §6 · p. 10
• LIGHT is a cognitive-inspired framework that combines episodic, working, and scratchpad memories to improve LLM performance.Source: §6 · p. 10

Long-Context Large Language Models

• Context expansion is enabled by attention-cost innovations, positional encodings, long-context training, and inference optimizations such as PagedAttention and KV-cache compression (Kwon et al., 2023; Li et al., 2024).Source: A DETAILED RELATED WORK · p. 17

Long-Term Memory Methods

• Long-term memory work spans recurrent and compressed architectures, state-space models, external memory, context compression, and retrieval-augmented generation (Rae et al., 2019; Gu & Dao, 2023; Borgeaud et al., 2022; Chevalier et al., 2023; Lewis et al., 2020).Source: A DETAILED RELATED WORK · p. 17

Contribution

• The paper proposes a novel retrieval-augmented method that shows substantial improvements over baselines in long-memory evaluation.Source: A DETAILED RELATED WORK · p. 17

Long-Term Memory Benchmarks

• Benchmarks range from naturalistic but topically limited DialSim to brief-session MSC and LoCoMo, while LongMemEval tests information extraction and temporal reasoning across 500 questions (Kim et al., 2024a; Xu et al., 2021; Maharana et al., 2024; Wu et al., 2024).Source: A DETAILED RELATED WORK · p. 17
• Existing benchmarks lack coherence, domain breadth, memory abilities, and bidirectional interaction; ours spans diverse domains up to 10M tokens and adds contradiction resolution, event ordering, and instruction following.The comparison is summarized in Table 2.Source: A DETAILED RELATED WORK · pp. 17–18, Table 2 · p. 18

B.1 DATASET STATISTICS

• Table 3 summarizes average user and assistant message counts per chat across chat sizes.The table defines these as average utterance counts from each speaker.Source: Table 3 · p. 18, §B.1 · p. 18
• The dataset statistics include how often the assistant asked questions answered by users and how many follow-up questions users asked.Both are reported as averages per chat in each chat size.Source: Table 3 · p. 18
• Table 3 also reports average dialogue turns per chat for each chat size.Turns are defined as the total number of dialogue turns.Source: Table 3 · p. 18, §B.1 · p. 18

Conversation Quality Evaluation

• Two annotators rated each generated conversation on a 1–5 Likert scale for Coherence and Flow, Dialogue Realism, and Complexity and Depth.The dimensions assessed continuity and thread consistency, natural interactions and believable progression, and interconnected topics with increasing difficulty.Source: §B.2 · pp. 18–19, Table 4 · p. 19
• To cover long conversations, annotators read 25 turns at the beginning, middle, and near the end, plus randomly selected turns between these regions.They checked progress against the conversation plan, follow-up questions, and possible hallucinations or unexpected behaviors.Source: §B.2 · p. 19

Inter-Annotator Agreement

• On 20 independently assessed conversations, Cohen’s κ was 0.7044 for Coherence and Flow, 0.7391 for Dialogue Realism, and 0.7849 for Complexity and Depth.The annotators completed this agreement assessment before the full evaluation.Source: §B.2 · p. 19

B.3.1 DOMAIN COVERAGE OF THE DATASET

• 100 multi-turn chats span 19 technical and non-technical domains, including coding, mathematics, health, therapy, lifestyle, and trip planning.Distinct chat titles define each dialogue’s thematic scope; Table 5 lists the domains and titles.Source: §B.3.1 · pp. 19–20, Table 5 · p. 20

B.3.2 CONVERSATION PLAN GENERATION

• Conversation plans combine a domain-and-topic seed, evolving narratives, user profiles, relationships, and a timeline into chronologically anchored stages.Plans use sub-plans with narrative bullet points and time anchors; a second-stage augmentation adds points targeting contradiction resolution, knowledge update, and instruction following.Source: §B.3.2 · pp. 22–23
• For 10M-token conversations, the authors create ten interlocking plans using either sequential life-stage expansion or hierarchical decomposition of the main seed.Plans maintain core relationships and use prior-plan summaries and future seeds to preserve continuity and respect topical and temporal boundaries.Source: §B.3.2 · pp. 22–23

B.3.3 USER UTTERANCE GENERATION / B.4 USER UTTERANCE GENERATION HYPERPARAMETERS

• User questions are generated from contiguous batches of plan bullets, conditioned on earlier dialogue context to maintain continuity while limiting repetition.Batch count K and questions per batch I are manually varied by domain and chat length; programming and mathematics prompts incorporate domain-specific interactions.Source: §B.3.3 · pp. 23–24, Table 6 · p. 27

B.3.4 ASSISTANT UTTERANCE GENERATION

• Assistant responses use recent dialogue turns and compressed summaries of older context, with threshold-controlled loops for assistant questions and user follow-ups.Both thresholds are set to two to prevent unbounded exchanges; the resulting dialogue is described as bidirectional, context-rich, and clarification-capable.Source: §B.3.4 · p. 24

B.3.5 ALGORITHMS

• The algorithms formalize the generation pipeline from conversation plans and batched user questions through iterative assistant answers and follow-up exchanges.The answer-generation procedure applies separate limits to assistant counter-questions and user follow-ups.Source: §B.3.4 · p. 25, §B.3.5 · p. 26

B.5 CREATED PROBING QUESTIONS DISTRIBUTION

• The probing-question distribution analysis assigns each question’s supporting evidence to one or more of ten equal conversation segments.Table 7 reports the segment distributions by conversation length; the supplied passage does not provide its values.Source: §B.5 · p. 27, Table 7 · p. 27

B.6 MEMORY ABILITIES EXAMPLES / 1. Abstention / 2. Contradiction Resolution

• Examples illustrate abstention when a conversation lacks requested information and contradiction resolution when the user gives conflicting accounts of real-estate event attendance.The contradiction example asks which of two statements—never attending such events or attending a Turkey rental-market webinar—is correct.Source: §B.6 · p. 28, §1 · p. 28, §2 · p. 28

3. Event Ordering (reasoning about chronological sequence)

• The event-ordering probe asks for ten stages in the user’s property-investment focus, from agent preparation through commercial-property considerations.The ideal answer lists the stages in sequence, including financial evaluation, financing, contractor management, permits, renovation, viewing, negotiation, and completion.Source: §3 · p. 28

4. Information Extraction (recalling factual details)

• $50,000 is the initial investment capital recalled by the information-extraction probe.Source: §4 · p. 28

5. Instruction Following (adhering to explicit user constraints)

• The instruction-following probe expects an upcoming-quarter budget answer with itemized costs, category breakdowns, and detailed cost analysis.Source: §5 · p. 28

6. Information Update (revising prior facts when updated)

• $60,000 is the updated initial-capital answer tested by the information-update probe.Source: §6 · p. 28

7. Multi-hop Reasoning (integrating evidence across turns)

• The multi-hop probe asks how many banks were considered for mortgage options and expects Halkbank and Ziraat Bank.Source: §7 · p. 28

8. Preference Following (adapting to evolving user preferences)

• The preference-following probe expects a structured property-comparison process based on data collection, measurable factors, and methodical analysis.Source: §8 · p. 29

9. Summarization (abstraction and compression of content)

• The summarization probe asks for the investment journey, including its $50,000 starting budget, property and management considerations, and financing plans.The ideal answer also covers market conditions, renovation planning, property-type tradeoffs, and financing through Halkbank and Ziraat Bank.Source: §9 · p. 29

10. Temporal Reasoning (reasoning about durations and timelines)

• 2 days separate the first property viewing with Mehmet Yilmaz on March 25 and the last scheduled viewing on March 27.Source: §10 · p. 29

C.1 ABLATION STUDY

• All ablation experiments use Qwen2.5-32B-AWQ as the base model.Source: §C.1 · p. 30
• The experiments evaluate the contribution of individual components in the proposed module.Source: §C.1 · p. 30
• Table 8 covers removal of retrieval, scratchpad, working memory, and noise filtering across conversation lengths of 100K–10M.Source: Table 8 · p. 30

C.2 RETRIEVAL BUDGET

• Retrieval-depth experiments compare K ∈ {5, 10, 15, 20}, alongside a dense retriever versus sparse SPLADE.Table 9 presents retrieval-depth performance across conversation lengths from 100K to 10M and memory abilities; the supplied passages do not report its cell values.Source: §C.2 · p. 30, Table 9 · p. 31
• 1.7% at 100K, 0.7% at 500K, and 0.8% at 1M are SPLADE gains, while 10M shows a 0.7% drop versus dense retrieval.Across conversation lengths, sparse retrieval provides a modest average improvement.Source: §C.2 · p. 31, Figure 5 · p. 32, Table 10 · p. 32

C.3 EFFECT OF EMBEDDING MODEL CHOICE

• 2.08% at 1M tokens and 16.37% at 10M tokens are LIGHT’s additional improvements when switching to bge-large with GPT-4.1-nano.The comparison uses bge-small versus bge-large embeddings.Source: Table 11 · p. 33, §C.3 · p. 33
• LIGHT consistently outperforms the RAG baseline under both bge-small and bge-large embedding configurations.The experiments keep the reader fixed to GPT-4.1-nano.Source: Table 11 · p. 33, §C.3 · p. 33

C.4 EFFECT OF INDEXING SETUP

• The indexing experiment switches LIGHT from IndexFlatIP to IndexHNSWFlat with GPT-4.1-nano as reader.The supplied passages identify the comparison but do not provide its result values.Source: §C.4 · pp. 33–34
• Table 12 reports LIGHT’s indexing-setup performance across conversation lengths from 100K to 10M and ten memory abilities.Its caption names two indexing configurations but provides no cell values.Source: Table 12 · p. 34

C.5 SUPPLEMENTARY BASELINE EVALUATION

• LIGHT consistently outperforms ReadAgent across 100K, 500K, 1M, and 10M-token conversations.The comparison covers ten memory abilities.Source: §C.5 · p. 34, Table 13 · p. 35

D NUGGET DESIGN

• The nugget-design section illustrates how probing questions yield evaluation nuggets for each memory ability.Source: D NUGGET DESIGN · p. 36

1. Abstention; 2. Contradiction Resolution

• Abstention rubrics require stating when information is absent, while contradiction-resolution rubrics require identifying both conflicting claims and requesting clarification.Source: §1 · p. 36, §2 · p. 36

3. Event Ordering

• Event-ordering rubrics assess whether the model lists events or topics in chronological order.The example asks for ten items and checks their sequence.Source: §3 · pp. 36–37

4. Information Extraction; 6. Knowledge Update

• Information-extraction rubrics check queried facts, while knowledge-update rubrics check that answers reflect changed values.Examples use an initial-capital fact and its updated value.Source: §4 · p. 37, §6 · p. 38

5. Instruction Following; 8. Preference Following

• Instruction-following and preference-following rubrics assess compliance with explicit response requirements and user-stated preferences.Examples test detailed budget breakdowns and methodical, data-driven property comparisons.Source: §5 · p. 38, §8 · p. 39

7. Multi-hop Reasoning

• Multi-hop reasoning rubrics check aggregation or comparison across sessions by scoring required intermediate or combined facts.The example asks how many banks were considered and names both.Source: §7 · p. 38

9. Summarization; 10. Temporal Reasoning

• Summarization rubrics assess coverage of required content, while temporal-reasoning rubrics assess durations and timeline relations.Examples cover a rental-property journey and the two-day interval between March 25 and March 27.Source: §9 · p. 39, §10 · p. 40

Coding-domain example

• The example centers on automating social media posts with Python, with a theme of scheduling across platforms and subtopics spanning APIs, scheduling, media, and error handling.The section notes that the narratives and subtopic list are shown only in part.Source: E EXAMPLES FROM DIFFERENT COMPONENTS… · p. 41

User profile

• The example supplies a user profile for John Brooks, describing him as a diligent, welcoming community mentor with a practical and orderly personality.The section says it also provides social relationships, but these passages do not specify them.Source: E EXAMPLES FROM DIFFERENT COMPONENTS… · p. 41

Conversation plan

• The dated conversation plan combines technical milestones with persistent user instructions and a logical contradiction, then tracks implementation changes across four subplans.Examples include a Twitter-account contradiction, OAuth and API work, retry logic, and scheduling latency improvements.Source: E EXAMPLES FROM DIFFERENT COMPONENTS… · p. 42

Generated dialogues

• Generated dialogue excerpts show the user refining a project plan with a request for small-scale testing and the assistant incorporating a dedicated testing phase.The section presents the exchanges as truncated exemplars of naturalistic follow-up turns.Source: E EXAMPLES FROM DIFFERENT COMPONENTS… · p. 44
• Later dialogue turns move from sharing code for Instagram engagement metrics to requesting Docker deployment guidance for an AWS EC2 instance.The assistant’s code review suggests schema safeguards, while the deployment response recommends pulling and running a container.Source: E EXAMPLES FROM DIFFERENT COMPONENTS… · pp. 44–45

Information Extraction

• For Information Extraction, LIGHT with the scratchpad recovers exact tool versions and the user’s city, country, and age, while LIGHT without it cannot answer either probe.The scratchpad consolidates scattered entity and version mentions.Source: F CASE STUDY · pp. 46–47

Instruction Following

• For Instruction Following, LIGHT with the scratchpad reports mentoring 3 junior editors, honoring the retained instruction to include the number of team members.Without the scratchpad, the model reports no recent mentoring activities.Source: F CASE STUDY · pp. 46–47

Knowledge Update

• For Knowledge Update, LIGHT with the scratchpad recalls the reduced memory figure of around 140MB per session and the extended April 5, 2024 deadline, avoiding stale answers.Without the scratchpad, it gives an under-100MB estimate and an April 1 deadline, respectively.Source: F CASE STUDY · p. 47

Temporal Reasoning

• For Temporal Reasoning, LIGHT with the scratchpad calculates 5 days between the sourdough events and 18 days between the survey and meetup, using normalized date anchors.Without the scratchpad, it answers 37 days and 28 days, respectively.Source: F CASE STUDY · p. 47

Case study takeaways

• Across the four case-study abilities, removing the scratchpad consistently causes failures that the full model avoids, while the scratchpad complements working and episodic memory.The paper characterizes it as a high-utility semantic scaffold for robust long-context behavior.Source: F CASE STUDY · p. 47

G QUALITATIVE ERROR ANALYSIS

• Missing context prevents correct extraction, multi-hop reasoning, instruction and preference following, and complete summaries; absent contradiction evidence also prevents conflict detection.For abstention, LIGHT may answer when the conversation contains no relevant information; in other tasks, missing required facts or instructions blocks a correct response.Source: G QUALITATIVE ERROR ANALYSIS · pp. 48–49
• Even when evidence is present, LIGHT may substitute similar facts, favor one side of a contradiction, omit details, or fail to use facts, instructions, or preferences.For abstention, similar entities, dates, or concepts can prompt an answer instead of abstention; contradiction errors can reflect position or frequency bias.Source: G QUALITATIVE ERROR ANALYSIS · pp. 48–49
• Retrieval can omit events or fail to preserve temporal order, leading to event-order and knowledge-update errors, while temporal reasoning also fails through date confusion or incorrect arithmetic.For knowledge updates, retrieved documents may omit the new value or present old and updated values in the wrong order.Source: G QUALITATIVE ERROR ANALYSIS · pp. 48–49

H PROMPTS

• The prompts select precise factual details for extraction and group related bullets across sessions to test aggregation, comparison, and multi-hop synthesis.The multi-session prompt prioritizes groups that connect multiple batches and support calculations, before-and-after comparisons, or synthesis across three or more batches.Source: H PROMPTS · pp. 50–51
• Other prompts pair original facts with their updates and pair events to test duration calculations, sequence, and temporal relationships, prioritizing some pairs across distant batches.The temporal prompt specifies that only absolute dates count as explicit time mentions.Source: H PROMPTS · pp. 51–52
• Temporal-reasoning prompts select bullet pairs for duration questions and add batch Time Anchors when one or both bullets lack explicit absolute dates.The prompt distinguishes explicit absolute dates from other time mentions and requires 8–10 sophisticated pairs.Source: H PROMPTS · p. 52
• Preference-following prompts select preference statements—not instructions—favoring explicit, reasoned, comparative, or recurring choices.They request only the specified number of bullets, prioritizing detailed preferences and clear reasoning.Source: H PROMPTS · pp. 52–53
• Event-order prompts group 8–12 or more related bullets across batches to test conversation mention order, prioritizing groups that begin in early batches.They target conversational chronology rather than the real-world chronology of events.Source: H PROMPTS · p. 53
• Contradiction-resolution prompts select pairs of logically impossible statements, emphasizing absolute claims and conflicts that cannot be explained away.Targeted patterns include never/always/only violations, timeline or location impossibilities, and capability or relationship contradictions.Source: H PROMPTS · p. 54
• Summarization prompts group 8–12 related bullets by content across categories into complete entity histories or end-to-end process arcs, prioritizing early-batch foundations.The prompts direct analysis of bullet content and entities across all batches, rather than relying on category names.Source: H PROMPTS · pp. 54–55
• Summarization candidates are selected by searching entities across all batches and categories, then grouping complete mentions into rich, varied sets for synthesis questions.The prompt emphasizes interconnected factual and narrative information, cross-session condensation, and groups with different complexity levels.Source: H PROMPTS · p. 55
• Instruction-following candidates are user-labeled bullets with explicit behavioral, formatting, content, or process directives, with all such bullets selected from each batch.The prompt requests the full bullet text and identifies roughly 10 such bullets as a target.Source: H PROMPTS · pp. 55–56
• Information-extraction questions must indirectly test specific facts by requiring synthesis across conversation turns, avoid answer-revealing details, and match question perspective to the information source.The prompt also requires hard, multi-detail questions and tracking every chat containing relevant information.Source: H PROMPTS · pp. 56–57
• Multi-session reasoning questions target complex synthesis across 4+ sessions, prohibit explicit temporal references, and must use user-provided information with relevant session references.The prompt calls for advanced reasoning such as aggregation, pattern recognition, comparison, and multi-hop analysis.Source: H PROMPTS · p. 57
• Knowledge-update questions ask for the current version rather than outdated information, while withholding hints about the prior and updated facts.The prompt supplies original and updated information from different sessions as input.Source: H PROMPTS · p. 58
• Knowledge-update probes ask for the current fact without mentioning a change, while checking that the answer reflects the latest version rather than outdated information.The prompt also requires tracking chat IDs containing the original and updated facts, while excluding those IDs from the answer.Source: H PROMPTS · pp. 58–59
• Temporal probes require hard, multi-step reasoning across conversation sessions, such as duration calculations, temporal patterns, inference, and synthesis, without explicit dates or times.Questions should span at least three sessions and use advanced temporal reasoning rather than simple lookup.Source: H PROMPTS · pp. 59–60
• Abstention probes ask natural, plan-related questions about plausible but unavailable information, including absent topics or missing specifics, at varying difficulty levels.The prompt prioritizes questions from early batches and requests 8–15 questions.Source: H PROMPTS · pp. 60–61
• Preference-following probes test whether the model respects a user preference while making recommendations, using neutral questions that do not reveal the preference or its criteria.The prompt requires extracting preference-related terms to exclude, and tracking all chat IDs where the preference was mentioned or demonstrated.Source: H PROMPTS · p. 61
• Event-ordering questions test recall of when topics were mentioned in conversation, not when events occurred, and require hard sequence reconstruction without event or time hints.The prompt sets a hard target of 8–10 mentions or comparably complex patterns and requires identifying the source chat IDs for each mention.Source: H PROMPTS · pp. 61–62
• Contradiction probes ask a routine, broad question about conflicting information so the model must detect the conflict and request clarification instead of answering definitively.The prompt also requires tracking the chat IDs containing each contradictory statement.Source: H PROMPTS · pp. 62–63
• Advanced summarization questions require synthesizing 8–10 bullet points across 4+ sessions into a coherent summary, while naming only the overarching topic and not hinting at its contents.The prompt calls for integrating multiple narrative threads and prioritizing important details.Source: H PROMPTS · pp. 63–64
• Instruction-following examples are selected from plans by identifying explicit user directives, including behavioral, conditional, formatting, content, and process instructions.Preference probes ask whether the model applies a user’s stated preference when offering recommendations or advice, with clear expectations and measurable signs of compliance.Source: H PROMPTS · pp. 61, 64
• Instruction-following probes select User Instruction bullets and assess whether responses comply with the specified requirement.The prompt directs selection of roughly ten such bullets across batches.Source: H PROMPTS · pp. 64–65
• The evaluator checks responses against rubric criteria, requiring responsiveness, accepting semantic equivalents, and ignoring style unless explicitly mandated.It scores compliance as 1.0, 0.5, or 0.0; compound criteria are assessed for whether all, some, or none of their elements are met.Source: H PROMPTS · p. 65
• Each conversation batch must include numerous concrete, verifiable details, with at least 5–7 detail categories and at least one verifiable detail per bullet.The prompt prioritizes exact numbers, dates, named locations, product names, decisions, outcomes, preferences, and quantifiable results to support factual questions.Source: H PROMPTS · p. 67
• Plans must advance chronologically from an initial story foundation, building on prior elements and showing measurable progression across batches.The prompt also calls for baselines in early batches, metric changes in middle batches, and cumulative before-and-after results later.Source: H PROMPTS · pp. 67–68
• Relationship continuity requires specific interactions and outcomes, while character preferences and past events remain consistent across batches.The prompt specifies relationship stages from introduction through maturation and calls for two to three concrete relationship bullets per batch.Source: H PROMPTS · p. 67
• Coding prompts require 5–7 concrete, verifiable technical details per batch and continuity across development phases, from basic implementation to debugging and optimization.Examples of required detail types include exact numbers, dates, named technologies, error messages, performance metrics, and configuration details.Source: H PROMPTS · pp. 68–69
• Mathematics prompts require specific equations, values, methods, and results while sequencing batches to show chronological learning progression.The distribution includes problem-solving activities, measurable learning status, a temporal anchor, and decisions between methods; batches should shift focus with the learning phase.Source: H PROMPTS · pp. 69–70
• A plan-editing prompt preserves original bullets and adds exactly three in fixed positions: an information update, a user instruction, and a logical contradiction.The update must change a value explicitly present in the plan, and the contradiction targets a completed action or permanent state.Source: H PROMPTS · pp. 71–72
• The topic-clustering prompt generates 10 chronologically coherent topics, retaining the seed as Topic 0 and assigning Topics 1–9 to different life domains rather than extending its story.The topics should form a natural life progression, with experiences leading to growth in other areas.Source: H PROMPTS · p. 72
• Topics should link causally across different life domains while matching the user’s circumstances and values, with realistic 1–2-month durations and non-overlapping timelines.The guidance also calls for broad topics with multiple decision points, substantial subtopics, ongoing processes, and cross-domain effects.Source: H PROMPTS · pp. 72–73
• The hierarchical seed prompt divides a main action into preparation, core-action, and integration phases, requiring unique subtopics that progress from planning through execution to reflection.It also calls for extracting the action’s dates and allocating realistic preparation and integration time.Source: H PROMPTS · p. 74
• Batch plans should preserve established character and relationship details, build on previous events, and keep temporal references within the assigned timeline.The prompt specifies first-person narration and chronological temporal anchors for each batch.Source: H PROMPTS · pp. 75–76
• Each batch should include concrete details and conflicts with clear stakes, while story progression tracks measurable changes from early baselines to later outcomes.The prompt calls for 2–3 situations per batch with clear stakes and examples include financial, time-management, relationship, professional, and values-based conflicts.Source: H PROMPTS · pp. 75–76
• Plan prompts require concrete, internally consistent details and enforce timeline boundaries, while carrying established facts and progression forward from previous plans.Dates must remain within the supplied timeline; prompts also call for prices, percentages, durations, causal triggers, and references to past events.Source: H PROMPTS · pp. 76–78
• Plans must form a coherent narrative across batches, with exactly 30 first-person bullets, a temporal anchor, and a staged progression from setup through long-term impacts.Each batch also allocates space for relationship developments, situations, events, and decisions.Source: H PROMPTS · p. 77
• General question prompts require questions to use details from current focus bullets only, cover each detail once, avoid prior-batch content, and vary natural conversational strategies.Context references are allowed only to understand details appearing in the current bullets; the prompt also specifies question types such as advice, clarification, and emotional processing.Source: H PROMPTS · p. 78
• Coding-question prompts prohibit code for time-anchor or personal-introduction bullets, but require substantial, production-style code and varied debugging, review, implementation, and optimization tasks for technical bullets.The prompt specifies that 85% of coding questions include 20–60+ lines of code and assigns target shares of 40%, 25%, 20%, and 15% to the four task patterns, respectively.Source: H PROMPTS · pp. 79–80
• The coding prompt requires exactly the requested number of realistic questions, uses every focused detail, and calls for substantial code in 85% of questions.Questions follow bullet order and one of four coding patterns: implementation, debugging, code review, or optimization.Source: H PROMPTS · p. 80
• The math prompt distinguishes time-anchor and personal-introduction bullets from mathematical ones, while requiring complete math work for the latter.Its four behavioral patterns allocate 40% to completely stuck, 30% to partially stuck, 20% to answer verification, and 10% to conceptual confusion; 80% of questions must include substantial work.Source: H PROMPTS · pp. 81–82
• A follow-up classifier returns YES only when the assistant asks for a specific user response, and NO for generic or rhetorical prompts.A separate follow-up decision prompt says to ask when information, clarification, or practical guidance is missing, and not when the user can proceed.Source: H PROMPTS · pp. 82–83
• The simulated-user prompt calls for short, natural replies that reflect personal experiences, consistent personality, informal speech, and genuine emotion.The reply should respond to the assistant’s question while using the supplied conversation history and plan context.Source: H PROMPTS · p. 83
• The assistant-answer prompt prioritizes established plan information, direct concise answers, and consistency with prior context and the conversation’s tone.It also says to avoid unnecessary questions and to answer in English.Source: H PROMPTS · p. 84
• The user-answer prompt asks the model to stay in character, respond only as the user, maintain consistency with established plans, and fill missing details in a storyline-consistent way.Source: H PROMPTS · p. 84
• A follow-up-question prompt directs the user LLM to scan current and previous batches, avoid repeated topics, and ask one natural 10–20-word question tied to the plan.It also encourages conversational reactions and tailoring the question to the current exchange.Source: H PROMPTS · p. 84
• The scratchpad creation prompt analyzes the latest exchange with conversation history to identify facts, updates, and implications, then outputs only a structured Extracted Facts block.Its example includes project context, budget, deadline, requested case studies, and a constraint against naming direct competitors.Source: H PROMPTS · p. 85
• The scratchpad summarization prompt clusters related content, removes redundancy, prioritizes relevant details, and preserves newer values and important chronology within a token limit.It organizes compressed content into categories such as entities, decisions, workflows, preferences, dates, and actionable items.Source: H PROMPTS · p. 86
• A scratchpad noise filter classifies whether a text chunk is needed to answer a query, returning only yes or no.Source: H PROMPTS · p. 86
• The RAG answer prompt requires direct, concise answers using only the supplied context, without relying on internal knowledge or adding explanation.Source: H PROMPTS · pp. 86–87