PaperlayerView exact revision on arXiv

Source-linked AI summary

LINE Conversation History Retrieval for Personal Memory RAG: Evaluating Search Representations and Hybrid Retrieval

Akito Hattori

arXiv:2608.27809v1cs.IR

TL;DR

Personal Memory RAG needs retrieval methods for private conversation histories whose wording and evidence may be difficult to match directly. This study evaluates representations and retrievers on one user’s LINE history, finding a stronger selected hybrid point estimate while identifying limitations from distributed evidence and exploratory evaluation.

  • Problem

    Search-representation comparisons for short, everyday, private conversation histories remain limited, although such histories contain user-specific context relevant to Personal Memory RAG.

  • Method

    The study segments one user’s LINE history into chunks, compares raw_text, summary, and embedding_text with BM25, dense, and hybrid retrieval, and evaluates 100 questions.

  • Results

    Recall@5 = 0.697 for embedding_text_bm25 plus embedding_text_vector at beta = 0.45, exceeding embedding_text_bm25 by 0.113 with a selection-conditioned 95% CI of [0.048, 0.184].

  • Takeaways & Limitations

    Lower point estimates on aggregate questions and unrecovered evidence at Top 20 indicate that distributed evidence may require retrieval-unit or memory-structure redesign beyond simply increasing K.

  • Takeaways & Limitations

    This exploratory case study uses one user’s LINE history and does not generalize to other users, platforms, or languages.

Abstract

from arXiv · show

As an initial step toward personal memory retrieval-augmented generation (RAG) for large language models (LLMs), this study presents a retrieval-only case study over one user's LINE conversation history. We segmented 358,896 messages into 22,329 temporally coherent chunks and constructed three search representations: raw_text, a generated summary, and embedding_text, which combines a summary with a raw-text excerpt and other fixed text. We compared BM25, dense vector retrieval, and linear hybrid retrieval on 100 evaluation questions verified by a single annotator. Among individual retrievers, embedding_text_bm25 achieved the highest point estimate, with Recall@5 of 0.584. We then explored six retriever pairings and 21 weights, for 126 configurations on the same evaluation set. The selected combination of embedding_text_bm25 and embedding_text_vector at beta = 0.45 achieved Recall@5 = 0.697, MRR@5 = 0.595, and nDCG@5 = 0.575. Its Recall@5 exceeded that of embedding_text_bm25 by 0.113, with a question-level paired percentile-bootstrap 95% confidence interval of [0.048, 0.184]. This interval is conditional on fixing the configuration selected on the same 100 questions and does not account for uncertainty from configuration selection or weight search. The difference from a summary-based hybrid at beta = 0.50 was 0.050, with a 95% confidence interval of [-0.013, 0.115], so no clear difference could be established. The 17 aggregate questions also yielded lower point estimates than the other question types, suggesting that flat chunk-level retrieval struggles when evidence is distributed across multiple times and conversations. This evaluation is an exploratory single-user, single-annotator study conducted on the same question set used for configuration search; it does not evaluate final answer generation or generalization to unseen questions.

1. Introduction

Personal Memory RAG treats personal conversation history as external memory that can help LLMs answer questions using user-specific context. This study introduces a retrieval-only LINE case study comparing retrieval representations and methods, with failure analysis focused on distributed evidence.

  • Personal Memory RAG uses searchable personal conversation history to provide LLMs with context about experiences, commitments, schedules, relationships, and emotions.
  • LINE histories are difficult to retrieve because short, elliptical messages contain context-dependent wording, errors, abbreviations, stickers, images, and URLs.
  • The study compares original text, generated summaries, and a summary-plus-raw-excerpt search representation for personal conversation retrieval.
  • The evaluation focuses on ranking conversation chunks that support answers rather than generating final answers with an LLM.
  • The study asks how representations and retrievers perform, which questions fail, and what additional design elements Personal Memory RAG may require.
  • The study contributes a retrieval-only evaluation environment, comparisons across representations and retrievers, and failure analysis identifying difficulty with distributed evidence.

2. Related Work

Personal conversation retrieval extends RAG and personal-information research by treating private dialogue history as external memory. Because dialogue requires lexical and semantic matching across contextual, time-dependent units, the study compares sparse, dense, and hybrid retrieval.

  • RAG adds retrieved external knowledge to an LLM’s input context, while this study evaluates only the retrieval stage preceding answer generation.
  • Comparisons of search representations for short, everyday, private conversation histories such as LINE remain limited.
  • Conversation retrieval must account for speakers, chronology, and surrounding context because individual chat messages are often short and elliptical.
  • Sparse, Dense, and Hybrid Retrieval: The study segments conversation history into temporally coherent units rather than treating each utterance as an independent retrieval unit.
  • Sparse, Dense, and Hybrid Retrieval: BM25 supports lexical matching, dense retrieval captures paraphrases and semantic similarity, and hybrid retrieval combines them.

3. Dataset and Search Representations

The dataset contains 358,896 messages from one user’s LINE history, segmented into 22,329 temporally bounded chunks. Each chunk receives raw-text, summary, and embedding_text representations with different lexical and contextual properties.

  • LINE History Dataset: 358,896 messages from one user’s LINE history were segmented into 22,329 chunks, each used as a retrieval unit.Raw messages, real names, source-file names, and specific conversation content were not released; examples are synthetic.
  • LINE History Dataset: Messages were read from date-keyed UTF-8 JSON files with time, sender, type, text, and source_order preserved.
  • LINE History Dataset: Sessions were split after gaps greater than 180 minutes, then chunked with windows of at most 42 messages and a stride of 36.Windows did not cross source or session boundaries, and adjacent windows overlapped by six messages.
  • Search Representations: raw_text preserves timestamps, speakers, vocabulary, proper names, and colloquial expressions but can be difficult to interpret in isolation.Its mean length was 635 Japanese characters, with a maximum of 10,031.
  • Search Representations: summary is an AI-generated natural-language summary produced from up to the first 7,000 Japanese characters of raw_text.Summary length averaged 273 Japanese characters, but unsupported additions and omissions were not audited.
  • Search Representations: embedding_text concatenates a fixed identifying sentence, the generated summary, a Japanese original-excerpt label, and up to 4,000 Japanese characters of raw_text.The raw excerpt was appended programmatically rather than generated by the language model.

4. Problem Formulation and Evaluation

The study evaluates Top-5 retrieval of gold evidence chunks for 100 questions using graded relevance and macro-averaged ranking metrics. Questions and labels were constructed through candidate recovery and annotation procedures with several scope and selection caveats.

  • 4.1 Retrieval-only Setting: The retrieval-only task ranks 22,329 conversation chunks that can support answers to natural-language questions, rather than generating final answers.
  • 4.1 Retrieval-only Setting: Top-k evaluation uses both the ordered ranking and set membership of retrieved chunks to measure recovery of gold evidence.
  • 4.2 Evaluation Questions: Candidate questions and evidence were generated with NotebookLM, then matched against the local database to recover candidate chunks; candidates were not accepted directly as gold labels.
  • 4.3 Gold Annotation: Candidate recovery and annotation were not fully independent or blind, and overlapping windows counted as separate gold chunks, potentially affecting Recall denominators.
  • 4.2 Evaluation Questions: The 100 retained questions comprise 34 named_entity, 49 contextual, and 17 aggregate questions covering specific entities, surrounding context, and information across conversations or time points.
  • 4.3 Gold Annotation: Relevance 2 denotes primary gold, relevance 1 supporting evidence, and relevance 0 non-supporting chunks; annotations contain 144 primary-gold and 45 secondary-gold chunks.
  • 4.4 Evaluation Metrics: Recall@5, MRR@5, and nDCG@5 are the primary metrics, with Top-5 chosen as a plausible number of chunks to pass to an LLM.
  • 4.4 Evaluation Metrics: Metrics are macro-averaged over 100 questions, while the hybrid configuration was selected from 126 configurations on those same questions, making its bootstrap interval selection-conditioned.The interval excludes uncertainty from configuration selection, weight search, and multiple comparisons.

5. Retrieval Methods

The study compares lexical, dense, and hybrid retrieval across three representations of temporally chunked LINE conversations. Hybrid retrieval combines normalized BM25 and vector rankings from up to 400 candidate chunks, while the implemented configurations omit raw_text_vector.

  • Representations and individual retrievers: Three representations—raw_text, summary, and embedding_text—were evaluated with BM25 and dense vector retrieval.The resulting BM25 retrievers were named raw_text_bm25, summary_bm25, and embedding_text_bm25.
  • BM25 retrieval: BM25 used a shared regex tokenizer with lowercasing, Japanese stopwords, and no morphological analysis or character n-grams.This makes the results specific to the regex-tokenized BM25 baseline.
  • Dense vector retrieval: Dense retrieval embedded questions and documents with text-embedding-3-small and ranked all 22,329 chunks by normalized vector inner product.Queries were embedded verbatim without rewriting, instruction prefixes, or Unicode normalization.
  • Design scope: raw_text_vector was not evaluated, so the study is not a complete representation × retrieval-method factorial experiment.The embedding_text_vector representation nevertheless included excerpts derived from raw_text.
  • Hybrid retrieval: Hybrid retrieval min–max normalized each retriever’s Top-200 scores, unioned their candidates, and linearly combined the normalized scores.The BM25 weight beta ranged from 0.00 to 1.00 in increments of 0.05 across six pairings, yielding 126 configurations.
  • Candidate-generation ceiling: The selected embedding_text pair had Top-200 oracle-union Recall of 0.941, but candidate generation still excluded some gold chunks.For this pair, embedding_text_bm25 and embedding_text_vector individually reached 0.859 and 0.888, respectively.

6. Results

Embedding-text BM25 had the strongest individual-retriever point estimate, while the selected embedding-text hybrid achieved the highest explored hybrid result. Comparisons were exploratory because configuration selection and evaluation used the same 100 questions, and aggregate questions had lower point estimates.

  • 6.1 Overall Single Retriever Results: 0.584 Recall@5 was the highest individual-retriever point estimate, achieved by embedding_text_bm25.The other reported individual Recall@5 values were 0.427 for raw_text_bm25, 0.463 for summary_bm25, 0.541 for summary_vector, and 0.549 for embedding_text_vector.
  • 6.2 Hybrid Retrieval Results: 0.697 Recall@5 was achieved by embedding_text_bm25 + embedding_text_vector at beta = 0.45, the highest point estimate among 126 explored configurations.The same configuration achieved MRR@5 = 0.595 and nDCG@5 = 0.575.
  • 6.3 Statistical Reliability: +0.113 Recall@5 was the selected hybrid’s difference from embedding_text_bm25, with a 95% CI of [0.048, 0.184].The interval was computed with 5,000 paired-bootstrap iterations after fixing the selected configuration.
  • 6.3 Statistical Reliability: +0.050 was the difference from the summary hybrid at beta = 0.50, with a 95% CI of [−0.013, 0.115].Because this interval included 0, no clear difference was concluded for that comparison.
  • 6.4 Performance by Question Type: 0.345 Recall@5 was obtained on aggregate questions, below 0.789 on named_entity and 0.755 on contextual questions.Only 17 aggregate questions were available, and their confidence interval was wide: [0.159, 0.551].

7. Failure Analysis

Failure analysis shows that the selected hybrid misses primary-gold evidence most often for aggregate questions, where relevant evidence is distributed across chunks, dates, people, and sources. Increasing the retrieval cutoff recovers some failures, but persistent misses may require changes to the retrieval unit or memory structure.

  • 7.1 Top-5 Failures of the Selected Best Hybrid: 22 of 100 questions had no primary-gold chunk in the selected hybrid's Top-5 results.The selected hybrid was embedding_text_bm25 plus embedding_text_vector at β = 0.45.
  • 7.1 Top-5 Failures of the Selected Best Hybrid: Aggregate questions had the highest failure rate, with 8 of 17 lacking primary gold in the Top 5.Recall@5 was 0.345 for aggregate questions, compared with 0.789 for named_entity and 0.755 for contextual questions.
  • 7.2 Aggregate Question Analysis: Aggregate questions distribute evidence across chunks, dates, people, and sources, challenging flat chunk-level retrieval.Such retrieval may find locally relevant chunks without ranking the multiple evidence pieces needed for an answer near the top.
  • 7.3 Auxiliary Failure Analysis with the Summary Hybrid: Of nine aggregate questions failing at Top 5 under the summary hybrid, two were recovered at Top 10 and four at Top 20.The summary hybrid was used for this auxiliary diagnostic analysis and failed at Top 5 on 30 of 100 questions.
  • 7.3 Auxiliary Failure Analysis with the Summary Hybrid: Failures remaining at Top 20 may require redesigning the retrieval unit or memory structure rather than simply increasing K.The paper suggests hierarchical summaries, person- or project-specific timelines, query decomposition, and multi-hop retrieval, but evaluates none of them.
  • 7.4 Retriever Characteristics: Embedding_text BM25 and vector retrieval succeed on different questions, while 45 questions are retrieved successfully by both and 19 by neither.The pattern is consistent with complementary retrieval behavior, but the hybrid improvement cannot be causally attributed to complementarity alone because normalization, candidate union, and score distributions also change.

8. Discussion

The discussion interprets embedding_text and hybrid retrieval as promising but unresolved design choices for Personal Memory RAG, especially when evidence spans conversations. Results remain exploratory because components, configurations, and final answer generation were not fully isolated or evaluated.

  • Embedding_text combines a natural-language summary with a raw excerpt, preserving meaning alongside proper names, dates, event names, and colloquial expressions.
  • The experiment compares complete search representations rather than isolating the effects of the fixed prefix, summary, and raw excerpt.
  • The results suggest that search representation and retriever choice interact, but do not establish embedding_text as consistently superior across retrieval methods.
  • 8.1 Implications of Hybrid Retrieval: The best explored hybrid used beta = 0.45, while Recall@5 remained 0.687 at beta = 0.50; however, the weight was selected in-sample.
  • 8.2 Toward Personal Memory RAG: Aggregate questions had lower performance, indicating difficulty for flat chunk retrieval when evidence is distributed across conversations, dates, people, and chunks.
  • 8.2 Toward Personal Memory RAG: Potential extensions include query expansion, query decomposition, reranking, multi-hop retrieval, and hierarchical representations, but these were not evaluated.
  • 8.2 Toward Personal Memory RAG: The retrieval-only study does not assess the accuracy, evidence faithfulness, or usefulness of final LLM-generated answers.

9. Limitations and Ethical Considerations

The study’s conclusions are constrained by its single-user, single-annotator, in-sample evaluation and several reproducibility and privacy limitations. Its data-processing practices also leave unresolved ethical concerns regarding third-party communications.

  • 9.1 Limitations: The case study uses one user’s LINE history, so its results cannot be generalized to other users, platforms, or languages.
  • 9.1 Limitations: Question generation may be biased toward formats or salient topics that NotebookLM readily produces and may not represent questions asked in actual use.
  • 9.1 Limitations: Gold labels came from one annotator and a 910-candidate pool, without exhaustive coverage checks or inter-annotator agreement measurement.
  • 9.1 Limitations: The same 100 questions were used for hybrid and beta search, making reported performance and intervals conditional on configuration selection.
  • 9.1 Limitations: The BM25 baseline lacks morphological analysis and character n-grams, potentially disadvantaging colloquial raw_text and confounding representation comparisons.
  • 9.2 Ethics and Privacy: Embedding_text retains up to 4,000 Japanese characters of raw text, and the study provides no formal anonymization, differential privacy, or privacy guarantee.
  • 9.2 Ethics and Privacy: Messages were submitted to external services without explicit consent from conversation partners or institutional ethics review.
  • 9.2 Ethics and Privacy: Public release excludes raw and individual evaluation content, but the study did not establish dedicated encryption, retention, or verified deletion procedures.

10. Conclusion

This study evaluated retrieval as an initial foundation for Personal Memory RAG using one user’s LINE history. The selected hybrid improved retrieval point estimates, while aggregate questions and component attribution remained important limitations.

  • 358,896 messages from one user were segmented into 22,329 chunks and evaluated for retrieval of answer-supporting conversation chunks across 100 questions.
  • Recall@5 = 0.697 for the selected embedding_text_bm25 + embedding_text_vector hybrid at β = 0.45, exceeding the best individual retriever by 0.113.The selection-conditioned 95% CI for the difference was [0.048, 0.184].
  • The selected hybrid’s higher point estimate than the summary hybrid did not establish a clear difference because its confidence interval included 0.
  • Aggregate questions showed lower performance, indicating difficulty retrieving evidence distributed across multiple times and conversations with flat chunk-level retrieval alone.Candidate improvements include topic-level memory, timelines, multi-hop retrieval, and query decomposition.
  • Future work should extend this retrieval foundation into a complete Personal Memory RAG pipeline that supplies retrieved chunks to an LLM for answer generation.Suggested additions include evidence-attributed answers, long-term memory updates, hierarchical memory representations, and LLM-based query generation.

A. Summary Generation Prompt

The study generated Japanese summaries of LINE-message chunks for later semantic search, preserving supported personal and conversational context. Each summary used a specified length target and was paired with an excerpt of the original text.

  • Summary generation used gpt-4o-mini with max_output_tokens=700, a 90-second timeout, and a 0.2-second call interval.Temperature, top-p, seed, and an explicit retry count were not set.
  • The prompt instructed the model to summarize original LINE messages in Japanese for later semantic search.
  • The summaries were instructed to preserve proper names, relationships, topics, emotions, commitments, events, and romance, school, and work contexts.
  • The instructions required avoiding excessive speculation and including only information supported by the original text.
  • The prompt required approximately 200–500 Japanese characters and appended up to the first 7,000 Japanese characters of raw_text after a blank line.

B. Experimental Environment and Provenance

The experimental record included an audited execution-environment table and preserved a question-ID-level metric CSV for the final evaluation. Exact candidate-question generation could not be reproduced because key generation details were not saved.

  • Table 10 is identified as the audited execution environment.
  • Candidate-question generation cannot be reproduced exactly because the generation prompt and provider-side model version were not saved.
  • A supplementary question-ID-level metric CSV was saved for the final 100 questions, but its raw text and question text were removed.
Loading 2608.27809v1…