PaperlayerView exact revision on arXiv

Source-linked AI summary

ConvMem: Convolutional Memory for Long-Context Reasoning

Hongming Zhang, Zhaozhen Gu, Fengshuo Bai, Ming Hao, Qingyang Zhang, Yuanyuan Wang, Shiyang Tang, Yanna Wang, Bo Xu

arXiv:2609.10441v1cs.AIcs.CL

TL;DR

Long-context reasoning is limited by attention costs, sequential memory latency, and RL-related overfitting. ConvMem addresses these issues with a training-free hierarchical convolution that parallelizes segment and reasoning-thread processing, and it outperforms training-free baselines while generalizing better than RL-trained models on out-of-distribution tasks.

  • Problem

    Long-context systems face scalability and latency problems, while sequential memory agents introduce unparallelizable execution and RL-driven overfitting to specific datasets.

  • Method

    ConvMem uses a frozen query-prompted LLM as a semantic convolutional kernel, hierarchically summarizing segments with configurable strides, skip connections, and multi-kernel channels.

  • Results

    ConvMem outperforms training-free baselines and shows superior out-of-distribution generalization compared with RL-trained models on RULER-HotpotQA and RULER-2WikiMultiHopQA.

  • Takeaways & Limitations

    The framework provides a parallelizable alternative to sequential memory reasoning while reducing cumulative error propagation through a logarithmic reasoning structure.

  • Takeaways & Limitations

    ConvMem uses more total computation than linear scanning and depends on correct query decomposition; flawed decomposition can leave subsequent kernels with incomplete premises.

Abstract

from arXiv · show

While Large Language Models (LLMs) have demonstrated impressive capabilities, they often struggle with extremely long contexts due to fixed context limits. To address this, sequential approaches like MemAgent extend the effective context by reading text in segments and iteratively updating a fixed-size memory. However, this sequential paradigm suffers from high latency and requires costly reinforcement learning (RL) training, which can lead to overfitting on specific datasets. To overcome these limitations, we propose ConvMem, a training-free, highly parallelizable framework that reformulates long-context reasoning as a hierarchical convolution. Inspired by CNNs, ConvMem treats an LLM prompted with a specific query as a convolutional kernel. This kernel summarizes text segments hierarchically, shortening the reasoning path from a linear chain into a logarithmic tree. Specifically, ConvMem integrates \textit{Configurable Strides} and \textit{Skip Connections} to ensure robust evidence capture and propagation, while employing \textit{Multi-Kernel Convolution} to decompose complex queries into disentangled semantic channels. This design not only mitigates error accumulation but also enables massive parallelization across both text segments and reasoning threads. Experiments on RULER-HotpotQA and RULER-2WikiMultiHopQA demonstrate that ConvMem outperforms training-free baselines and avoids the risk of overfitting to parametric priors often observed in RL-trained models on out-of-distribution tasks.

1 Introduction

ConvMem addresses the latency, scalability, and overfitting limitations of sequential long-context memory agents with a training-free hierarchical convolutional framework. Its logarithmic reasoning structure and architectural mechanisms support parallel processing, evidence preservation, and improved generalization.

  • Long-context LLM reasoning faces quadratic attention costs, performance degradation with increasing context length, and practical deployment bottlenecks.
  • Sequential memory agents reduce complexity to O(N) but impose an unparallelizable temporal bottleneck and can overfit through reinforcement-learning-based memory policies.
  • ConvMem treats a frozen, query-prompted LLM as a semantic convolutional kernel that processes text segments hierarchically and in parallel.
  • The hierarchy converts a linear reasoning chain into an O(log N) tree, shortening inference paths and mitigating cumulative error propagation.
  • Configurable strides, skip connections, and multi-kernel convolution improve evidence capture, preserve high-confidence details, and separate intertwined reasoning threads.
  • On RULER-HotpotQA and RULER-2WikiMultiHopQA, ConvMem outperforms training-free baselines and generalizes better than RL-trained models on out-of-distribution tasks.

2 ConvMem for Long-Context Reasoning

ConvMem reformulates long-context reasoning as a hierarchical, multi-channel convolution using a frozen LLM as a semantic kernel. Overlapping scans, skip connections, and parallel reasoning channels preserve evidence while reducing the reasoning depth to logarithmic scaling.

  • 2.2 The Semantic Convolutional Kernel: ConvMem treats a frozen, query-prompted LLM as a semantic convolutional kernel that transforms local text segments into summaries and relevance scores.The kernel performs information extraction and compression while identifying segments relevant to the query.
  • 2.2 The Semantic Convolutional Kernel: The semantic convolution slides this kernel across document segments, concatenating local summaries into a global summary and recording relevance scores for critical evidence.The operation extends local text-to-text processing to the full document.
  • 2.3.1 Configurable Strides: Overlapping windows with stride S < W preserve contextual continuity and scan tokens from multiple positions, improving evidence recall and robustness to kernel stochasticity.Tokens near segment boundaries are reprocessed in neighboring windows, while multi-view scanning cross-verifies information.
  • 2.3.2 Skip Connections: Skip connections append segments assigned critical relevance scores directly to the final reasoning stage, preserving precise raw evidence alongside hierarchical summaries.This residual path is designed to retain fine-grained details that summarization may compress away.
  • 2.3.3 Multi-Kernel Convolution: Multi-kernel convolution decomposes a complex query into distinct sub-questions, processes them in parallel through separate semantic channels, and keeps their reasoning paths disentangled.Channel-specific buffers and summaries preserve isolation between orthogonal reasoning threads until final aggregation.
  • 2.4 Hierarchical Parallel Inference Workflow: ConvMem recursively summarizes segments across layers while parallelizing independent segments and kernels, reducing the reasoning path to O(log N) depth rather than sequential execution.The resulting latency depends theoretically on tree depth, and the authors state that million-token contexts can have latency comparable to short contexts.

3 Experiments

ConvMem is evaluated against standard LLMs, training-free memory methods, and RL-trained specialists on in-distribution and out-of-distribution long-context benchmarks. It outperforms training-free baselines, generalizes more robustly than RL specialists, and benefits from its architectural mechanisms.

  • Experimental Setup: ConvMem is compared with standard Qwen2.5 models, retrieval and training-free memory methods, and RL-trained memory specialists.The baselines span vanilla language models, RAG-BM25, non-RL memory agents, and RL-optimized agents.
  • Experimental Setup: Evaluations use RULER-HotpotQA as an in-distribution benchmark and RULER-2WikiMultiHopQA as an unseen out-of-distribution testbed.The latter is designed to distinguish reasoning from memorization because it was not used to train MemAgent.
  • Main Results: ConvMem establishes a new state-of-the-art among training-free methods and demonstrates superior robustness compared to RL-based specialists across varying context lengths.Table 1 reports F1 and Sub-EM across contexts from 28k to 896k.
  • Main Results: RAG struggles with multi-hop reasoning, while sequential non-RL agents suffer cumulative forgetting and lose access to early evidence.ConvMem instead preserves related facts through hierarchical aggregation and parallel channels.
  • Main Results: RL-based specialists show instability: Mem-α loses F1 through verbose outputs, while MemAgent collapses on the unseen RULER-2WikiMultiHopQA dataset.These observations are presented as evidence that RL models may prioritize parametric memorization over in-context reasoning.
  • Scalability: ConvMem yields immediate performance gains across 7B–72B backbones without adaptation costs.This supports its model-agnostic scaling behavior.
  • Ablation Study: Fivefold over-scanning optimizes the coverage–noise trade-off, skip connections preserve fine-grained details, and multi-kernel convolution improves multi-hop accuracy.The ablations attribute these effects respectively to boundary coverage, bypassing compression loss, and disentangling reasoning threads.

4 Related Work

Prior long-context methods use architectural extrapolation, retrieval or sequential memory, and reinforcement learning, but face limitations in context fidelity, parallelism, latency, or generalization. ConvMem instead uses a training-free, tree-structured convolutional topology.

  • ConvMem Design Context: Figure 5 links query decomposition and kernel size to ConvMem’s design: multiple channels reduce thread interference, while W = 8000 balances context coherence and signal density.These choices correspond to multi-kernel convolution and kernel-size sensitivity.
  • Architectural and Attention Methods: Long-context research includes positional extrapolation, efficient attention, and KV-cache compression, yet native long-context models can exhibit lost-in-the-middle degradation.These approaches target context extension or computational efficiency but do not remove the cited reasoning degradation.
  • Memory-Augmented Systems: RAG reduces context by retrieving top-k chunks but often fails on multi-hop tasks because isolated fragments lack global connectivity.Sequential memory systems preserve a stream through iterative updates but introduce sequential dependencies.
  • Memory-Augmented Systems: Sequential memory processing has O(N) dependency structure that prohibits parallelization and causes high inference latency.ConvMem addresses this with a CNN-inspired tree topology of O(log N).
  • Reinforcement-Learning Memory: RL-based memory policies can improve in-domain performance but incur training costs and remain susceptible to dataset overfitting.ConvMem instead leverages pretrained instruction-following without reward optimization.

5 Conclusion

The paper presents ConvMem as a training-free, hierarchical multi-channel convolutional framework for long-context reasoning. It replaces linear dependency chains with logarithmic trees and reports competitive standard-benchmark performance plus stronger out-of-distribution generalization than RL-trained specialists.

  • Conclusion: ConvMem reformulates long-context reasoning as a hierarchical, multi-channel convolutional process using multi-kernel convolution, configurable strides, and skip connections.The framework is designed to combine computational efficiency with reasoning fidelity.
  • Conclusion: The method transforms an O(N) linear dependency chain into an O(log N) tree, mitigating cumulative error propagation.The conclusion connects the topology change to shorter information propagation paths.
  • Conclusion: ConvMem achieves competitive standard-benchmark performance and superior out-of-distribution generalization compared with RL-trained specialists.The conclusion also reports that RL-based agents tend to overfit dataset artifacts.

6 Limitations

The paper identifies two limitations: ConvMem reduces latency through parallelization at the cost of higher total computation, and its effectiveness depends on correct query decomposition.

  • Limitations: ConvMem’s logarithmic latency comes with higher total computational cost than linear scanning methods.This creates a token-consumption versus latency trade-off.
  • Limitations: Parallel convolution depends on the backbone model correctly disentangling complex queries into orthogonal sub-questions.If decomposition is flawed, subsequent kernels may operate on incomplete premises.

7 Ethical Considerations

ConvMem is training-free but requires multiple text passes, increasing inference computation. Its performance may also inherit biases from the frozen LLMs used as kernels, while experiments use public data only.

  • Multiple text passes increase inference computation despite eliminating energy consumption associated with model training.
  • Potential biases may stem from the underlying frozen LLMs used as convolutional kernels.
  • All datasets used in the study are publicly available, and no private data was involved.

A Dataset Details

The experiments use multi-hop QA datasets and construct long-context benchmarks by inserting relevant evidence among distractor documents. The novel 2WikiMultiHopQA benchmark provides an out-of-distribution test intended to distinguish reasoning from memorization and reduce dataset-specific ambiguity.

  • Source datasets: HotpotQA requires reasoning across multiple Wikipedia documents and provides supporting sentences for explainable answers.
  • Source datasets: 2WikiMultiHopQA adds structured Wikidata-based questions and comprehensive evidence paths, reducing shortcut reasoning relative to HotpotQA.
  • Long-context benchmarks: RULER-HotpotQA injects distractor paragraphs to create contexts ranging from 28k to 896k tokens and filters questions answerable from parametric knowledge alone.
  • Long-context benchmarks: RULER-2WikiMultiHopQA is a novel out-of-distribution benchmark built by inserting relevant paragraphs into noise documents while preserving multi-hop logic.
  • Long-context benchmarks: The 2WikiMultiHopQA construction is intended to reduce HotpotQA ambiguity and measure retrieval and reasoning rather than dataset-specific biases.

B Implementation Details

The implementation uses a reproducible, shared hardware and backbone setup, with hierarchical recursion and bounded query decomposition supporting ConvMem’s long-context processing. ConvMem’s key configuration uses overlapping scans and dynamically determined semantic channels.

  • Infrastructure: Experiments use NVIDIA A800 GPUs, vLLM with PagedAttention and continuous batching, and bfloat16 Qwen2.5-32B-Instruct inference.
  • Hierarchical recursion: Hierarchical summarization adds layers until intermediate summaries fit within kernel size W, producing depths of L ∈[2, 3] up to 128k tokens and L ∈[3, 4] up to 1M.
  • Query decomposition: Query decomposition limits ConvMem to Cmax = 5 sub-questions, while the model typically generates 2-4 sub-questions.
  • Query decomposition: Parsing failures fall back to a default relevant state r = 1 to prevent information loss.
  • Evaluation setup: All baselines use the same Qwen2.5-32B-Instruct backbone and hardware, while RAG-BM25 and MemAgent follow specified retrieval and original-paper settings.
  • Hyperparameters: ConvMem’s detailed hyperparameters are reported in Table 2.

C Full Experimental Results

The paper reports full performance comparisons across context lengths and datasets using four metrics, alongside an analysis of baseline failure modes. These results are intended to assess sequential-architecture limitations and overfitting tendencies in RL-trained agents.

  • Performance comparison: Performance is compared across context lengths from 28k to 896k tokens using four metrics in Table 3.
  • Performance comparison: Table 3 covers RULER-HotpotQA and 2WikiMultiHopQA with F1, EM, Sub-EM, and ACCL metrics.
  • Failure-mode analysis: The experimental analysis examines baseline failure modes as evidence concerning sequential architectures and RL-trained agents’ overfitting tendencies.

D.1 Sequential Information Loss in Recurrent Architectures

Sequential memory agents can lose early evidence before later context reveals its relevance, causing failures on multi-hop links and encouraging reliance on memorized priors. ConvMem addresses these weaknesses by processing segments in parallel and preserving context-grounded evidence.

  • Sequential Information Loss: Sequential agents may discard initially irrelevant information, making it unrecoverable when later context reveals its relevance.This forgetting bottleneck limits their ability to connect evidence distributed across a text stream.
  • Disconnected Evidence: ConvMem avoids the Hard Easy failure by processing distant segments in parallel channels.Parallel processing preserves the relationship between the actor and the episode despite their separation in the stream.
  • Disconnected Evidence: In the Hard Easy example, the sibling relationship appears early while the episode connection appears later, requiring a global view to answer correctly.The analysis identifies this separation as a failure mode for sequential agents.
  • Parametric Bias: RL-trained agents tend to overfit training distributions by prioritizing memorized parametric priors over provided context.This risk can make an agent favor a memorized answer even when the input evidence contradicts it.
  • Parametric Bias: In the Lev Yilmaz case, MemAgent follows the mistyped dataset label, whereas ConvMem extracts the context-supported name.The example illustrates reliance on parametric priors versus adherence to input evidence.

E.1 Dataset Ambiguity and Missing Evidence

The evaluation examples expose ambiguity, missing evidence, and answer variations that make strict matching unreliable. The section therefore motivates robust semantic metrics and describes query decomposition and relevance-aware memory updates used in ConvMem.

  • Dataset Ambiguity: Some synthetic benchmark queries are ambiguous or unanswerable from the provided context, indicating flaws in dataset generation.The Brown State Fishing Lake example omits the requested United States population while supplying only Brown County’s population.
  • Missing Evidence: Missing evidence can force models either to hallucinate from parametric knowledge or to answer with an irrelevant value from the context.The example contrasts the unavailable national population with the supplied county population of 9,984.
  • Evaluation Ambiguity: Exact Match penalizes valid answers that use aliases, omit middle names, or include explanatory context.Examples include Kelly Osbourne versus Kelly Lee Osbourne and complete-sentence answers for a yes/no question.
  • Evaluation Ambiguity: Sub-EM handles subset and verbosity variations, while ACCL captures semantic equivalence when lexical overlap metrics fail.The presidential-election example receives ACCL = 1.0 despite zero Sub-EM and Exact Match scores.
  • ConvMem Processing: ConvMem decomposes complex queries into independent subproblems and keywords before applying relevance-aware convolutional summarization.Its prompts classify segment relevance and update memory with information useful for answering the question.
Loading 2609.10441v1…