PaperlayerView exact revision on arXiv

Source-linked AI summary

Panini: Continual Learning in Token Space via Structured Memory

Shreyas Rajesh, Pavan Holur, Mehmet Yigit Turali, Chenda Duan, Vwani Roychowdhury

arXiv:2602.15156v1cs.AI

TL;DR

Existing non-parametric approaches store and retrieve external experience, but Panini structures documents as GSWs and traverses them with RICR for reasoning-grounded retrieval. Across six QA benchmarks, it reports the strongest average performance with substantially fewer answer-context tokens, while also evaluating open-source pipelines and abstention under missing evidence.

  • Problem

    Non-parametric continual learning must store growing experience externally while supporting accurate reasoning, efficient retrieval, and abstention when evidence is missing.

  • Method

    Panini builds entity- and event-aware GSW networks of QA pairs and uses RICR to decompose queries and follow reasoning chains through them.

  • Results

    56.1 average performance across six benchmarks, versus 53.3 for HippoRAG2 and 50.5 for dense retrieval, while using 2.2× fewer tokens than chunk retrieval and 5–30× fewer than structure-augmented and agentic methods.

  • Takeaways & Limitations

    GSW provides structured memory that supports Panini’s lightweight retrieval and can also improve an agentic system without retraining.

  • Takeaways & Limitations

    High-quality GSW construction remains more expensive with proprietary models and less reliable with smaller open-source models.

Abstract

from arXiv · show

Language models are increasingly used to reason over content they were not trained on, such as new documents, evolving knowledge, and user-specific data. A common approach is retrieval-augmented generation (RAG), which stores verbatim documents externally (as chunks) and retrieves only a relevant subset at inference time for an LLM to reason over. However, this results in inefficient usage of test-time compute (LLM repeatedly reasons over the same documents); moreover, chunk retrieval can inject irrelevant context that increases unsupported generation. We propose a human-like non-parametric continual learning framework, where the base model remains fixed, and learning occurs by integrating each new experience into an external semantic memory state that accumulates and consolidates itself continually. We present Panini, which realizes this by representing documents as Generative Semantic Workspaces (GSW) -- an entity- and event-aware network of question-answer (QA) pairs, sufficient for an LLM to reconstruct the experienced situations and mine latent knowledge via reasoning-grounded inference chains on the network. Given a query, Panini only traverses the continually-updated GSW (not the verbatim documents or chunks), and retrieves the most likely inference chains. Across six QA benchmarks, Panini achieves the highest average performance, 5%-7% higher than other competitive baselines, while using 2-30x fewer answer-context tokens, supports fully open-source pipelines, and reduces unsupported answers on curated unanswerable queries. The results show that efficient and accurate structuring of experiences at write time -- as achieved by the GSW framework -- yields both efficiency and reliability gains at read time. Code is available at https://github.com/roychowdhuryresearch/gsw-memory.

1. Introduction

Non-parametric continual learning keeps the base model fixed while structuring new experiences in external memory for efficient, accurate, and reliable question answering. Panini combines GSW structured memory with RICR chain-following retrieval and outperforms competitive baselines across the paper’s three criteria.

  • Motivation: Bounded context windows and long-context failures make in-context learning difficult to scale as stored information grows.The introduction highlights lost-in-the-middle and context rot as scaling problems.
  • Motivation: Non-parametric continual learning stores new documents and experiences externally while keeping the base model fixed.The framework is motivated by continually integrating experiences into memory rather than repeatedly re-injecting them into prompts.
  • Design criteria: An NPCL system should synthesize supported answers, retrieve efficiently as memory grows, and abstain when stored experience lacks evidence.These criteria distinguish continual learning from systems that merely retrieve passages and answer queries.
  • Prior approaches: RAG retrieves verbatim fragments, while structured approaches such as RAPTOR and GraphRAG emphasize compression or thematic summaries over linkage-based latent-knowledge retrieval.The related work contrasts passage retrieval and summarization-oriented representations with reasoning across structured linkages.
  • Panini: Panini writes documents as entity- and event-aware GSW networks of question–answer pairs and reads them with RICR, which follows reasoning-grounded inference chains.GSWs support reconstructing experienced situations and mining latent knowledge; RICR decomposes queries and traverses the network.
  • Panini: Panini combines structured memory with chain-following retrieval and reports stronger supported performance, inference-time efficiency, and reliable abstention than competitive baselines.The paper evaluates these three criteria through extensive QA benchmarking.

2. Panini

Panini represents experiences as structured Generative Semantic Workspaces and reads them through reasoning-grounded inference chains. Its retrieval pipeline decomposes queries, follows entity-linked QA pairs across hops, scores alternative chains, and supplies compact evidence for answer generation.

  • Structured Memory: A GSW is an entity- and event-aware semantic network of QA pairs that supports reconstructing situations and mining latent knowledge through inference chains.QA pairs attach questions and answers to event structures, making entities associated with an event explicit in atomic units.
  • Structured Memory: Panini indexes GSWs over entities and QA pairs, using matched entities as entry points while retaining compact QA evidence instead of passages or large graph neighborhoods.The dual indices combine sparse BM25 entity retrieval with semantic QA-pair retrieval before reranking.
  • RICR: At inference, a decomposition module rewrites each query into ordered atomic sub-questions whose later inputs are answers from preceding hops.The system can also create parallel sequences for queries requiring separate reasoning paths.
  • RICR: RICR expands candidate paths hop-by-hop, selecting QA answers that instantiate subsequent sub-questions and assembling the results into scored chains.Each hop resolves one atomic sub-question and provides the answer used to continue the same sequence.
  • RICR: Unlike agentic systems that repeatedly interleave decomposition and retrieval, Panini decomposes once and performs subsequent steps through non-parametric retrieval and scoring.The final answering LLM receives the top-ranked, de-duplicated chains as evidence.
  • RICR: Panini maintains B chains in parallel and prunes low-scoring alternatives, reducing the risk that an early retrieval error propagates through the complete chain.Chains are scored using the geometric mean of constituent relevance scores and retained with unique answers.

3. Experimental Setup

Panini is evaluated on factual-memory and multi-hop QA benchmarks, with additional curated tests for answering when evidence is missing. The setup compares chunk-based, structure-augmented, and agentic retrieval systems under common evaluation and prompting conditions.

  • Benchmarks: The evaluation targets factual memory and associativity, using single-hop and multi-hop question answering as a controlled testbed.Associativity requires discovering intermediate entities and composing information across documents.
  • Benchmarks: The multi-hop suite includes MuSiQue, 2WikiMultihopQA, HotpotQA, and LV-Eval, while NQ represents the single-hop setting.The benchmarks cover compositional reasoning, diverse multi-hop patterns, and supporting-fact reasoning across sources.
  • Reliability: Reliability is tested with MUSIQUE-PLATINUM and 2Wiki-PLATINUM, which separate answerable from genuinely unanswerable queries relative to available corpus evidence.Unanswerable cases arise from missing or insufficient evidence, annotation errors, or ambiguity.
  • Reliability: All methods use a common answer prompt, reporting answerable-subset performance and refusal accuracy on unanswerable queries.Ans measures EM/F1 on answerable examples, whereas Unans measures whether the model outputs the canonical non-answer token.
  • Metrics: Performance is reported with Exact Match and F1, while inference efficiency is measured by total prompt tokens supplied to the answer model.EM measures normalized string match and F1 measures token-level overlap with gold answers.
  • Baselines: Baselines span chunk retrieval, structure-augmented memories, and agentic multi-step retrieval, with GPT-4o-mini used as the default answer model.The comparison includes BM25 variants, RAPTOR, GraphRAG, HippoRAG, IRCoT, and Search-R1.

4. Results and Discussion

PANINI performs strongly across QA, efficiency, reliability, open-source, ablation, and transfer evaluations. Its structured GSW memory supports accurate, token-efficient retrieval and can improve another agentic system without retraining.

  • QA performance and efficiency: 56.1 average performance across six benchmarks exceeds HippoRAG2 at 53.3 and dense retrieval at 50.5.Gains are most pronounced on multi-hop tasks, where PANINI also exceeds agentic systems despite using one decomposition call.
  • QA performance and efficiency: 2.2× fewer tokens than chunk retrieval and 5–30× fewer than structure-augmented and agentic methods.PANINI conditions the answer model on short, targeted QA pairs.
  • Reliability: 79.8 answerable accuracy and 74.0 refusal accuracy show strong performance on both answerable and unanswerable queries.PANINI avoids the usual trade-off between missing answerable questions and hallucinating when evidence is insufficient.
  • Open-source pipeline: Fully open-source components preserve PANINI’s advantage over baselines and widen its gains on multi-hop tasks.The fully open pipeline fits on a single GPU, while performance degrades gracefully with noisier GSW extractions.
  • Ablations: Reducing beam width from 5 to 3 cuts token usage by about 25% with minimal accuracy impact.A wider search is more helpful for multi-hop questions.
  • GSW as reusable retrieval infrastructure: 47.3 to 49.4 average F1 improves Search-R1 when PANINI’s GSW index replaces BM25 chunk retrieval without retraining.The result supports GSW as reusable structured memory for lightweight chain retrieval and agentic systems.

5. Related Work

Related work spans parametric and non-parametric continual learning, retrieval and structured-memory systems, and multi-hop or agentic QA. PANINI shifts computation toward structured write-time representations and lightweight read-time chain retrieval rather than verbatim passage storage or repeated LLM reasoning.

  • Continual learning paradigms: Parametric continual learning updates model weights but risks catastrophic forgetting, expensive retraining, and conflicts with post-training alignment.Non-parametric methods keep the base model fixed and store new information externally.
  • Evaluation context: 56.06% highest average F1 across six QA benchmarks and 2–30× fewer generation-context tokens summarize PANINI’s evaluation advantages.These figures are reported in the performance and token-count tables.
  • Retrieval-augmented generation: PANINI writes structured, reusable abstractions rather than the retrievable text fragments used by standard RAG.This distinguishes its memory representation from conventional passage-level storage.
  • Continual learning paradigms: Parametric storage lacks native source attribution and selective retraction, whereas non-parametric memory maintains discrete, addressable records.Parametric knowledge editing can degrade after as few as ten updates.
  • Test-time training: Gradient-based test-time updates add computational overhead and serving latency, limiting practical deployment.This is a deployment boundary for test-time training and adaptation methods.
  • Multi-hop QA and agentic retrieval: Agentic multi-hop methods iteratively reason and retrieve but require multiple LLM calls, increasing token usage and latency.PANINI instead shifts computation to write-time structure and uses lightweight chain retrieval with minimal query-time LLM usage.
  • Structured memory and retrieval: RAPTOR and GraphRAG structure documents through summaries or entity relationships, but their representations target compression and thematic summarization rather than latent-knowledge reasoning.Structured approaches may trade retrieval precision and coverage against broad context windows or multiple retrieval rounds.
  • Structured memory and retrieval: PANINI resolves entities within documents and forms cross-document connections dynamically through RICR beam search at read time.Language-model-guided scoring is applied at each hop across separate GSW structures.

6. Concluding Remarks and Limitations

PANINI combines structured GSW memory with chain retrieval to answer questions efficiently and reliably, while identifying open directions for reconciliation, lower construction cost, and broader domains.

  • PANINI writes each document into reusable GSW memory and applies RICR for multi-hop reasoning over stored experience without iterative retrieval loops.GSW represents structured memory, while RICR follows reasoning chains at read time.
  • Across six QA benchmarks, PANINI achieves the strongest average performance while using 2–30× fewer answer-context tokens than competitive baselines.
  • On Platinum evaluation, PANINI maintains high answerable-question accuracy while substantially improving abstention accuracy when evidence is missing.
  • Limitations and Future Work: PANINI does not yet perform latent-link caching or experience-driven reconciliation, leaving recurring cross-document relations unintegrated.
  • Limitations and Future Work: High-quality GSW construction remains more expensive with proprietary models and less reliable with smaller open-source models.
  • Limitations and Future Work: Future work includes richer reconciliation, broader downstream retrieval infrastructure, and extensions beyond fact-centric QA to narrative-heavy and multimodal domains.

D. Open-Source Experiments

The open-source experiments evaluate PANINI along read-time and write-time axes, including a fully open-source configuration spanning memory construction, question decomposition, and answer generation.

  • Open-source experiments replace proprietary read-time components and separately vary the model used for GSW construction.
  • PANINI’s open-source evaluation is organized around complementary read-time and write-time operations.
  • The fully open-source pipeline runs every component—from memory construction to final answering—with open-source models.

D.1.1. EXPERIMENTAL SETUP AND IMPLEMENTATION DETAILS

The open-source setup specifies models for decomposition, GSW construction, retrieval, reranking, and reading, then evaluates PANINI across multi-hop QA benchmarks with different reader capacities.

  • Experimental Setup and Implementation Details: Question decomposition uses LoRA-finetuned Qwen3-8B or Qwen3-4B models, while answer generation evaluates Qwen3-4B, Qwen3-8B, and Qwen3-8B in thinking mode.
  • Experimental Setup and Implementation Details: The decomposition training data contains 1,780 manually reviewed examples grouped by 2-hop, 3-hop, and 4-hop structures.
  • Results: PANINI consistently outperforms lexical, dense, and graph-based baselines on multi-hop benchmarks with Qwen3-4B as the reader.
  • Results: With Qwen3-8B, PANINI achieves the highest scores on MuSiQue and 2Wiki and recovers most of the proprietary-answerer performance.

D.1.4. PLATINUM BENCHMARK EVALUATION

Platinum evaluation measures answerable accuracy and abstention on unanswerable questions, while additional experiments examine open-source construction, refinement, and RICR components.

  • Platinum Benchmark Evaluation: Platinum reports answerable accuracy and unanswerable accuracy, with higher unanswerable accuracy indicating improved abstention and lower hallucination rates.
  • Platinum Benchmark Evaluation: 75.60% average answerable accuracy for PANINI exceeds 66.98% for HippoRAG 2 on MuSiQue Platinum and 2Wiki Platinum.
  • Platinum Benchmark Evaluation: PANINI remains top-performing with thinking mode enabled, while method ordering remains largely unchanged.
  • Platinum Benchmark Evaluation: PANINI balances competitive unanswerable performance with substantially higher answerable accuracy than aggressive-abstention lexical baselines.
  • Result Analysis: GSW quality and downstream QA performance improve approximately monotonically with model capacity, with GPT-OSS-120B strongest overall among evaluated open-source constructors.
  • Result Analysis: A second GSW-construction pass repairs missing entities, relations, and malformed QA pairs, consistently improving downstream performance.
  • Ablation Studies: RICR ablations test question decomposition, dual-search retrieval, QA reranking, and comparisons with iterative agentic retrieval.

E.1.4. ABLATION: CHAIN-LEVEL SCORING

Chain-level scoring ranks reasoning beams using cumulative evidence quality, with the geometric-mean rule outperforming similarity-only, combined, and greedy alternatives. Beam-width ablations further show that narrower search can reduce tokens without harming accuracy until the beam becomes too narrow.

  • Chain-level scoring: The main cumulative scoring rule aggregates hop-wise reranker scores using a geometric mean that penalizes weak hops.Alternative scores use cosine similarity, linear interpolation, or only the most recent hop score.
  • Chain-level scoring: The cumulative scoring rule performs best, while similarity-only and combined variants underperform, and greedy last-hop selection remains competitive but falls short.
  • Beam width: Reducing beam width from B=5 to B=3 preserves accuracy while substantially reducing token usage.A single beam, B=1, reduces usage further but degrades performance, especially on multi-hop questions.
  • Beam width: B=1 lowers token usage but degrades performance, especially on multi-hop questions.

E.2. Continual Learning Ablation Studies

The continual-learning ablations test robustness to corpus growth and retrieval design. PANINI remains more stable as distractors accumulate, while structured QA and dual-index retrieval strengthen an agentic retrieval system.

  • Corpus expansion: PANINI degrades substantially less as the MuSiQue retrieval corpus expands from 4K passages toward approximately 12K passages.The evaluation questions’ gold-supporting passages are already present in the initial 4K subset, isolating expansion-related retrieval noise.
  • Corpus expansion: Embedding- and BM25-based baselines exhibit larger F1 drops as irrelevant content accumulates.
  • Retrieval variants: Dense Search-R1 retrieval consistently improves multi-hop performance over BM25, especially on MuSiQue, HotpotQA, and LV-Eval.
  • Retrieval variants: Replacing document chunks with PANINI’s extracted QA pairs yields competitive multi-hop results within the Search-R1 agentic framework.
  • Retrieval variants: Search-R1 achieves its strongest overall performance when combining entity-level retrieval with direct QA-pair similarity search.The dual-index design produces consistent gains across configurations.

F.2. Qualitative Error Analysis

PANINI’s qualitative errors arise from failures in GSW construction, QA connectivity, and question decomposition. These errors can remove evidence or send retrieval along an incorrect reasoning chain.

  • Recurring failure modes: Three recurring error modes are missing verb-phrase nodes, QA-pair construction errors, and imperfect question decomposition.
  • GSW construction: Omitting a key verb-phrase node leaves the relation partially captured and removes QA pairs needed for downstream chain-following retrieval.
  • QA-pair construction: A missing inverse QA link can prevent an entity from appearing as an answer, breaking later reasoning steps instantiated from retrieved entities.
  • Question decomposition: For long or ambiguity-prone queries, decomposition can generate sub-questions misaligned with available evidence and cause retrieval to follow an incorrect chain.

G. Computational Costs and Resources for Building the GSW

PANINI shifts substantial computation to one-time GSW construction while keeping read-time inference lightweight. Its construction prompt emphasizes comprehensive, bidirectional, entity-centered factual representation for multi-hop retrieval.

  • Write-time construction: $48.02 is the reported one-time GSW indexing cost for 11,656 MuSiQue passages using gpt-4.1-mini and the OpenAI Batch API.
  • Read-time inference: After indexing, PANINI’s read-time inference remains lightweight, with QA latency dominated by retrieval rather than repeated document processing.
  • Read-time inference: Question decomposition requires approximately 60 tokens per query and contributes negligibly to per-question runtime and cost.
  • GSW construction prompt: The GSW prompt instructs the system to extract all entities, including dates, places, titles, organizations, concepts, and answer-bearing values.
  • GSW construction prompt: The prompt requires bidirectional questions for each relationship and connectivity so every entity participates in at least one verb-phrase question set.
  • GSW construction prompt: The prompt also captures temporal information, aliases, universal object or content relations, and special relationships such as purpose and conditionality.
Loading 2602.15156v1…