InfiniteICL: Breaking the Limit of Context Window Size via Long Short-term Memory Transformation

1 Introduction

• ICL's finite context windows constrain long-context use, while attention costs and diminishing returns make expanded context capacity burdensome.The paper identifies quadratic attention complexity, linear KV-cache growth, and marginal accuracy gains on some extended-context tasks.Source: §1 · p. 1
• By treating context as short-term memory and parameters as long-term memory, the approach frames context transformation as a cognitive-inspired memory process.The framework also depicts transformations as refreshing the context window to support streaming input.Source: §1 · p. 1, Figure 1 · p. 1
• Context-to-parameter transformation reduces GPU memory use and may maintain performance across varied input lengths, while theoretically enabling continued context integration.The stated advantages are relative to conventional context-window extension strategies.Source: §1 · p. 2
• The framework elicits contextual knowledge, selects knowledge-critical pathways, and consolidates selected interactions into permanent parameter updates.Selection retains top-k interactions with the greatest perplexity discrepancy between context-aware and context-free generations.Source: §1 · p. 2
• Across fact recall, grounded reasoning, and 300-shot ICL, the method averages 103% performance recovery relative to full-context prompting, with scores of 84%, 117%, and 98%, respectively.On LongBench v2, sequential transformation outperforms full-context prompting up to 128K while using 0.4% of the original contexts.Source: §1 · p. 2

2 Related Work

• Context-compression approaches shorten input through summarization, token removal, trainable prompts, or KV reduction, but may lose context integrity as texts lengthen.Some approaches depend on downstream task performance or compress KV activations along the sequence length.Source: §2 · p. 2
• Existing context-distillation work often targets short contexts, specific domains, or specialized parameters and pretraining; this paper aims for a general approach evaluated across varied, long-context tasks.The cited examples include conversation history, sports documents, and task instructions.Source: §2 · pp. 2–3

3.1 Overview

• Human short-term and long-term memory provide the analogy for coordinating language-model context and parameters.The transformation is compared to hippocampal-cortical memory consolidation.Source: §3.1 · p. 3

3.2 Short-term Memory

• Increasing sequence length is associated with performance degradation, while token limits impose a capacity bottleneck on language-model context.The paper draws parallels to temporal decay and capacity limits in human memory.Source: §3.2 · p. 3

3.3 Long-term Memory

• Model parameters serve as a persistent knowledge repository that remains unchanged unless deliberately updated, without additional hardware burdens after fine-tuning.Source: §3.3 · p. 3

3.4 Memory Transformation

• Context transformation is compared with hippocampal-cortical consolidation, where transient traces are replayed, integrated, reinforced, pruned, or attenuated.The biological analogy includes sleep-like replay and dynamic restructuring of neuronal connections.Source: §3.4 · p. 3
• Context-to-parameter transformation reduces KV-cache memory and attention costs, may preserve performance across input lengths, and theoretically supports infinite context integration.Source: §3.4 · p. 3
• Learning parameters θ′ to match the teacher's context-conditioned distribution with the student's unconditional distribution is intended to internalize context without explicit attention to it at inference.The paper also views a Transformer's parameter set as the state of a meta recurrent neural network.Source: §3.4 · p. 3

4.1 Overview

• The three-phase framework elicits context knowledge, selects high-transfer examples, then aligns the student model with context-conditioned teacher outputs.The phases transform short-term context into long-term parameter memory.Source: §4.1 · p. 3

4.2 Context Knowledge Elicitation

• Elicitation generates diverse, skill-specific continuations, including question answering, summarization, multi-hop inference, and counterfactual reasoning.The process prompts for context-grounded queries and responses, forming transfer set T.Source: §4.2 · p. 4

4.3 Path Selection

• Path selection keeps the top-k query-response pairs per context with the largest perplexity discrepancy between teacher and student models.The selected pairs form refined subset Tk.Source: §4.3 · p. 4

4.4 Memory Consolidation

• Memory consolidation uses knowledge distillation to align the student with the teacher’s context-conditioned distribution over Tk.The paper explores hidden-state, logit-level, and sequence-level alignment, with task-specific trade-offs between retention and generalization.Source: §4.4 · p. 4

5 Evaluation Setup

• The evaluation separates single transformations, which isolate individual transformation efficacy, from sequential transformations, which assess viability under gradually received streaming inputs.Single- and multi-turn transformations provide the two evaluation settings.Source: §5 · p. 4

5.1 Single Transformation

• Single-transformation tests cover document QA, 300-shot classification, counterfactual and multi-hop knowledge updates, and book-text continuation, with contexts limited to 8K tokens.The datasets include Natural Questions, TREC Coarse, TREC Fine, NLU, CounterFact, MQuAKE, and PG19.Source: §5.1 · pp. 4–5

5.2 Sequential Transformations

• Sequential transformations use LongBench v2 across real-world task types and contexts ranging from 8K to 2M words.The benchmark includes QA, ICL, dialogue, code repository, and structured data understanding, with domains such as legal documents, scientific articles, and technical manuals.Source: §5.2 · p. 5

6.1 Baselines / 6.2 Metrics

• The evaluation uses full-context prompting as an upper bound, no-context performance as a lower bound, and task-specific EM, PPL, or accuracy metrics.Recovery rate compares method performance with the upper and lower bounds; full-context LongBench v2 evaluation also uses a 128K Llama3.1 model with middle-truncation as a practical upper bound.Source: §6.1 · p. 5, §6.2 · p. 6

6.3 Single Transformation Tasks

• 103% average performance recovery rate in single transformations, including 98% average recovery for many-shot ICL and 123% for knowledge updates.For document-based QA on Natural Questions, the method achieves 74% recovery while outperforming all baselines.Source: Table 1 · p. 6, §6.3 · pp. 6–7
• Across retention ratios from 0.7 to 0, InfiniteICL remains stable (68.8 →62.3; peak 72.3 at 0.1), while baseline methods degrade and collapse at extreme ratios.Figure 2 reports average model performance across retention ratios.Source: Figure 2 · p. 6, §6.3 · p. 7
• At N = 200, PPL-differential path selection peaks at 72.27, exceeding random selection by 2.36 points and KL-based selection by 1.67 points.Retaining the top 200 continuations, each 512 tokens long, yields the best reported result.Source: §6.3 · p. 7, Table 2 · p. 7
• Loss comparisons show task-dependent trade-offs: FKL exceeds MSE by 20% on NQ, while SeqKD scores 6.0 versus FKL’s 96.0 on Mquake.SeqKD leads on PG19 text generation (5.5 PPL versus FKL’s 22.7), but the authors select FKL as the most balanced choice.Source: §6.3 · p. 7, Table 3 · p. 7

6.4 Sequential Transformations Tasks

• InfiniteICL outperforms all other approaches in sequential transformations across overall performance, difficulty levels, and text-length categories, with substantial gains in Medium and Long contexts.The study evaluates these tasks using LongBench v2 and compares against 128K Context and context-compression methods.Source: §6.4 · pp. 7–8, Table 4 · p. 8
• As context length grows, baselines decline by 15.4% ± 12.8% over 8K–64K and collapse at 1M–2M, while InfiniteICL remains stable through contexts up to 2M.Some baselines show a 512K performance spike that the authors attribute to pretrained priors rather than genuine context use.Source: Figure 3 · p. 8, §6.4 · p. 8
• At 5K-token chunks, performance peaks at 49.11 with an average of 53 transformation turns; shorter chunks fragment integration, while 6K chunks reduce performance by 2.68 points.The authors identify around 5K tokens as an effective chunk range for an 8K-context-window model.Source: §6.4 · pp. 8–9, Table 5 · p. 9

• The framework transforms temporary context knowledge into permanent parameter updates through elicitation, path selection, and memory consolidation.Source: §7 · p. 9
• 103% average recovery rate in single transformation tasks surpasses full-context prompting.Source: §7 · p. 9
• At extreme compression ratios, the method maintains 62.25 performance while most baselines collapse to no-context performance.Source: §7 · p. 9
• Sequential transformation performance remains consistent across varying context lengths and task difficulties.Source: §7 · p. 9

• Experiments conducted primarily with Llama-3-8B-instruct may limit the generalizability of the findings.Source: Limitations · p. 9
• Gradient-based fine-tuning currently limits the framework’s computational efficiency.Source: Limitations · p. 9

• The hybrid prompt asks the model to generate queries covering summaries, details, inference, patterns, applications, and repetition of source text.It directs generation of 20 clearly formulated queries, each on a separate numbered line.Source: §A.1 · pp. 11–12
• The prompt supplies the context between explicit text-begin and text-end delimiters before asking the model to generate queries.Source: §A.1 · pp. 11–12
• Query and response sampling uses temperature 1, while testing uses temperature 0 for greedy decoding and reproducibility.Source: §A.1 · p. 12

A.2 loss variants

• Forward KL targets per-token distribution matching, while reverse KL prioritizes mode coverage; adaptive KL blends their weights based on divergence.Source: §A.2 · p. 12
• DPKD uses preference-driven reverse KL with length normalization, MSE aligns hidden states, and SeqKD applies sequence-level teacher forcing to sampled continuations.Source: §A.2 · p. 12

A.3 Impact of the number and length of the continuations.

• 72.27 is the peak performance for N = 200 and L = 512, while increasing the count to N = 300 at L = 512 yields 69.53.The authors describe a quality-quantity trade-off in which excessive continuation counts degrade information density.Source: §A.3 · p. 12, Table 6 · p. 12