Source-linked AI summary
A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts
Kuang-Huei Lee, Xinyun Chen, Hiroki Furuta, John Canny, Ian Fischer
TL;DR
LLMs struggle with explicit context limits and degraded performance on long inputs. ReadAgent uses prompted episode pagination, gist compression, and interactive look-up, outperforming baselines across three tasks while extending effective context length by up to 20×.
Problem
LLMs are limited by maximum context length and do not robustly consume increasingly long inputs.
Method
ReadAgent prompts an LLM to group text into pages, compress pages into contextualized gist memories, and look up original pages as needed.
Results
ReadAgent outperforms baselines across QuALITY, NarrativeQA, and QMSum, increasing effective context length by up to 20×.
Takeaways & Limitations
The results show that LLMs can generate useful compressed representations and reason interactively over them for long-context tasks.
Takeaways & Limitations
Gist memories elide details and may increase hallucination when tasks require information absent from the compressed memories.
Abstract
from arXiv · showhide
Current Large Language Models (LLMs) are not only limited to some maximum context length, but also are not able to robustly consume long inputs. To address these limitations, we propose ReadAgent, an LLM agent system that increases effective context length up to 20x in our experiments. Inspired by how humans interactively read long documents, we implement ReadAgent as a simple prompting system that uses the advanced language capabilities of LLMs to (1) decide what content to store together in a memory episode, (2) compress those memory episodes into short episodic memories called gist memories, and (3) take actions to look up passages in the original text if ReadAgent needs to remind itself of relevant details to complete a task. We evaluate ReadAgent against baselines using retrieval methods, using the original long contexts, and using the gist memories. These evaluations are performed on three long-document reading comprehension tasks: QuALITY, NarrativeQA, and QMSum. ReadAgent outperforms the baselines on all three tasks while extending the effective context window by 3.5-20x.
1. Introduction
ReadAgent addresses LLMs’ difficulty with long inputs by modeling human reading: it forms gist memories, then retrieves original passages when needed. Across three long-document tasks, it outperforms baselines while substantially increasing effective context length.
- LLMs face both explicit context-length limits and declining performance on increasingly long inputs.
- Human-inspired reading motivates storing durable gist information while looking up exact details from the original text when needed.
- ReadAgent divides long-text processing into episode pagination, memory gisting, and interactive look-up.Pagination creates pages from contiguous text; gisting compresses each page and preserves its context; look-up combines selected raw pages with the gists.
- ReadAgent evaluates against gist-only, full-text, and retrieval baselines on QuALITY, NarrativeQA, and QMSum.
- 12.97% LLM-rating improvement and 31.98% ROUGE-L improvement over the best retrieval baseline were achieved on NarrativeQA, with approximately 20× greater effective context length.
- 3.5× effective context length and 20.4% fewer consumed words were achieved on QuALITY compared with using full text.
2. Related Work
Prior approaches extend long-context handling through longer-context training, architectural changes, retrieval, or interactive agents. ReadAgent instead combines zero-shot prompting with contextualized gist-memory retrieval.
- Long-context LLM research explores longer-context training and Transformer attention architectures that reduce the need for long-context fine-tuning.
- Retrieval-augmented generation lets LLMs query task-relevant information from document collections or document pieces.
- ReadAgent implements retrieval by reasoning over contextualized gist memory using zero-shot LLM prompting.
- Interactive long-text agents use browsing, iterative prompting, or intermediate notes, but the cited methods cannot handle inputs exceeding the LLM context length.
- MemWalker interactively searches hierarchical summaries, whereas ReadAgent uses contextualized gist memories for long-document retrieval.
3. ReadAgent
ReadAgent builds gist memories from paginated text and interactively looks up original pages when task-specific details are needed. Its prompting-based pipeline supports parallel or sequential lookup while trading compression, detail preservation, and computational cost.
- 3.1. Gist Memory: ReadAgent constructs gist memory by grouping contiguous text into pages, shortening each page into a gist, and concatenating tagged gists.Pagination chooses natural paragraph-level pause points; memory gisting compresses each resulting page.
- 3.1. Gist Memory: Page size controls compression: larger chunks can remove more duplicated information but also tend to remove more details, creating a performance trade-off.The method adjusts minimum and maximum word limits during pagination to control page size.
- 3.2. Interactive Lookup: ReadAgent prompts the model to select relevant original pages for task-specific lookup, either in parallel or sequentially.Selected raw pages replace their corresponding gists, preserving the overall memory flow before final task solving.
- 3.2. Interactive Lookup: Sequential lookup exposes previously expanded pages before the next selection, while parallel lookup requests multiple pages in one interaction.Sequential lookup may provide more information for page selection but requires more model interactions and computational cost.
- 3.3. Computational Trade-offs and Scalability: Iterative pagination, gisting, and lookup add cost bounded linearly by a small factor, and gist creation can be amortized when contexts serve multiple tasks.On the QuALITY development set, up-to-2-page lookup reduced words consumed by 20.4% versus directly answering from the original articles.
4. Experiments
ReadAgent is evaluated on three long-document comprehension tasks using prompting-based pagination, gist memories, and interactive or retrieval-based look-up. It outperforms the baselines across these evaluations, although metric interpretation involves trade-offs in compression, response length, and evaluation measures.
- Evaluation setup: ReadAgent is evaluated on QuALITY, NarrativeQA, and QMSum without model training, using PaLM 2-L and comparisons with retrieval, full or truncated text, and gist-only baselines.The evaluation uses validation, test, and/or development sets to avoid overfitting system hyperparameters.
- QuALITY: ReadAgent-P reaches 86.91% accuracy and ReadAgent-S reaches 87.17% accuracy on QuALITY when looking up 1-6 pages.Performance increases as the maximum number of pages allowed for look-up increases.
- QuALITY: 3.5× as many tokens can fit in the context window after gisting, while ReadAgent with 1-2 page look-up outperforms all baselines including the full original text on QuALITY.The corresponding compression rate is 72.17%.
- NarrativeQA: ReadAgent outperforms all baselines across all NarrativeQA subsets.The paper reports separate results for Gutenberg and movie-script subsets because their text-length distributions differ.
- QMSum: ReadAgent-S substantially outperforms ReadAgent-P on QMSum, but this improvement costs up to six times as many retrieval-phase requests.Performance improves as compression rate decreases, so looking up more pages tends to improve results.
- Evaluation caveats: ROUGE comparisons are complicated because longer responses lower precision and F-Measures, while LLM ratings are harder to compare across rating models and prior work.The authors therefore consider LLM ratings more informative for comparisons between these runs.
- Analysis: Prompt-based retrieval performs better than neural retrieval when each method looks up one page, and LLM-selected pause points are beneficial compared with rule-based segmentation.The compression trade-off also shows that excessive initial compression can hurt ReadAgent accuracy despite look-ups.
5. Conclusion
The conclusion presents ReadAgent as an interactive prompting system that compresses long contexts and retrieves details as needed. Across the reported evaluations, it extends effective context length while outperforming conventional retrieval, but it remains bounded by gist-memory length and finite context capacity.
- Contribution: ReadAgent uses compressed textual representations and interactive reasoning to decide what information to retrieve for long-context tasks.The system is presented as a prompting-based approach rather than a trained or fine-tuned model.
- Results: Up to 20× effective context length is achieved while outperforming conventional retrieval techniques.The conclusion frames this as a result across the paper’s evaluations.
- Limitations: ReadAgent does not provide infinite context lengths and does not guarantee good performance when the gist memory itself is extremely long.The authors identify these as fundamental limitations requiring future work.
Impact Statement
ReadAgent inherits the impacts and risks of the LLMs it is built upon, while enabling attempts at problems current LLMs cannot tackle because of context-length limitations.
- ReadAgent naturally inherits the impacts and risks of the LLMs on which it is built.
- Its extended context handling makes it possible to attempt problems that current LLMs cannot tackle because of context-length limitations.
- The authors identify a plausible unstudied risk: gist memories may increase hallucination when omitted details are required.The model may generate missing details without indicating that it has done so.
B. Evaluation with GPT-3.5
The GPT-3.5 Turbo evaluation reuses the experimental setup and prompts used for PaLM 2-L. QuALITY fits within GPT-3.5 Turbo’s context, but the same overall trends remain.
- Table 7 evaluates the same setup described in Section 4.3.1 using GPT-3.5 Turbo rather than PaLM 2-L.
- GPT-3.5 Turbo has a context length of over 16,000 tokens, so QuALITY articles easily fit within context.
- The evaluation uses the PaLM 2-L prompts without specifically tuning them for GPT-3.5 Turbo.
- GPT-3.5 Turbo performs worse than PaLM 2-L on this task, while the same general trends hold.
- Table 7 reports QuALITY dev-set results for 230 documents and 2086 questions, with one run per experiment for cost considerations.
C. Pagination Hyperparameters
Pagination uses maximum- and minimum-word episode parameters, whose experiment-specific values are reported in Table 8.
- Maximum words and minimum words are the two episode-pagination hyperparameters described for the experiments.
- Table 8 reports the values of these pagination hyperparameters for each experiment in Section 4.
- The hyperparameter table concerns pagination settings rather than evaluation outcomes.
D. Case Study
The case studies show how ReadAgent combines gist-based global context with targeted page lookup, avoiding distractors that can mislead retrieval-based approaches. Additional web-navigation material describes HTML pagination, gist construction, interactive lookup, and Mind2Web evaluation.
- Distracting retrieval: For Dameri’s landing purpose, ReadAgent selects (D), whereas neural retrieval selects the animal-collection distractor (C).
- Distracting retrieval: ReadAgent’s pages 5 and 6, combined with gist-based global context, support the correct answer despite animal-related distractor pages.
- Incorrect retrieval: For what happened in government custody, ReadAgent chooses (B), while neural retrieval chooses (A).
- Incorrect retrieval: Neural retrieval’s inclusion of pages 0 and 1 confused the custody question despite the correct answer appearing on page 4 and in its gist.
- Incorrect retrieval: For the English-communication question, ReadAgent chooses (D), while neural retrieval chooses (C) after omitting the critical information on page 1.
- Web Navigation: For HTML, snippets from DOM-tree descendants serve as pages; gists are concatenated, and lookup uses task instructions, action history, and gists.
- Web Navigation: Mind2Web evaluates next-step web actions across cross-task, cross-website, and cross-domain test splits, alongside full HTML and retrieval baselines.
E.3. Results
ReadAgent performs strongly against web-navigation baselines, while gisting reduces input length and preserves gains even when truncation remains. The authors also report prompt and gisting-design observations that delimit current evaluation choices.
- Results: ReadAgent outperforms MindAct despite not using models trained on the web-navigation domain.The authors contrast this result with prior work reporting that state-of-the-art LLMs are generally weaker than domain-trained web-navigation systems.
- Results: 97.4% of cross-website gisted inputs fit within 8K tokens, compared with 51.5% of raw HTML inputs.Inputs exceeding the context limit are truncated, which can significantly affect performance.
- Results: ReadAgent gains over full context despite truncation because truncated gists and retrieved pages are more informative than truncated raw HTML for small-context LLMs.The retrieved snippets can remain large, reducing the compression rate substantially.
- Prompt design: Including the previous page in the pagination prompt benefited QMSum but not QuALITY, according to the reported experiments.The authors leave this design choice to future studies.
- Scope: The experiments use unconditional gisting, which may produce broader but less compressed memories than task-conditioned gisting.The authors note that conditional gisting may be preferred when the task is known in advance.
H. Comparing ReadAgent and MemWalker
The comparison presents ReadAgent as more reliable than the authors’ MemWalker reimplementation and less constrained by hierarchical traversal when related information is distant. NarrativeQA additionally requires controls for long gists and retrieved pages.
- Reliability: ReadAgent’s failure rate is mostly 0%, whereas the MemWalker reimplementation had an 11.7% search failure rate after retries.The reported MemWalker failure rate is similar to the authors’ cited 8.6% failure rate for the original system.
- Reliability: 66.73% on QuALITY for MemWalker was below full raw content at 85.83%, ReadAgent-P at 86.63%, ReadAgent-S at 86.88%, and BM25 Top-1 at 70.55%.The authors attribute part of the difference to MemWalker’s high search failure rate.
- Comparison: MemWalker’s hierarchical summaries make it difficult to reason over related distant information at the same granularity.Information preserved at higher levels may lack the detail needed to connect distant leaf segments.
- Comparison: ReadAgent reasons over gist memories while interacting directly with documents, whereas MemWalker reasons over traversal trajectories in a summary tree.The approaches therefore differ in both their interaction structure and reasoning object.
- NarrativeQA controls: NarrativeQA controls long contexts by iteratively merging pages and limiting retrieved pages so gists and lookups stay within the context window.Merged pages are re-gisted, while retrieved pages are ranked and added only when they fit.
J. Additional QMSum Results
Additional QMSum results report test-set evaluation and document the word-count distributions of original versus gisted transcripts. The table reports means and standard deviations across three PaLM 2-L runs.
- Context compression: Figure 8 compares word-count histograms for original QMSum transcripts and their gisted transcripts.The figure is described as showing the training-set distributions.
- Context compression: QMSum gisted transcripts are described as fitting entirely within PaLM 2-L’s context window when they are below 5,000 words.The supplied passage states the 5,000-word threshold for complete fit.
- Test results: Table 11 reports QMSum test results from 35 articles and 281 questions using PaLM 2-L.Results are means and standard deviations across 3 runs.
- Test results: Table 11 includes compression rate, number of lookups, and final response length alongside the QMSum test results.Standard deviations are omitted for compression rate and number of lookups because they were inconsequential.