Source-linked AI summary
Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data
Jinli Hu, Ross M. Clarke, Yichuan Zhang, José Miguel Hernández-Lobato
TL;DR
Deployed models generally cannot learn authentically from live interactions, so new information is carried in prompts and reread for each request. This paper generates low-rank updates to a shared network from live data and updates a belief over the generator’s latent code. Its experiments report that weight-carried data can beat prompt-carried data on long, noisy evidence, while the method remains bounded by the knowledge available from supplied data and its base model.
Problem
Deployed models keep weights frozen, leaving live interaction data outside the model or in context to be reread and discarded.
Source: §1 · p. 1Method
A compact generator maps live data to low-rank updates of a shared base network, while an online-updated belief over its latent code drives the generated weights.
Source: §3.1 · p. 6, §3.2 · p. 7, §3.3 · p. 9Results
The experiments report that weight-carried run-time data beats prompt-carried data on long, noisy evidence, and that belief-based routing improves as a conversation grows.
Source: §7 · p. 18Takeaways & Limitations
The design carries supplied run-time knowledge and behaviour in weights, amortizing compute, freeing context space, and persisting updates across turns.
Source: §7 · p. 17A compact generator does not inherit a large MoE’s stored knowledge; without relevant supplied facts or instructions, the model has only its base knowledge and default behaviour.
Source: §6 · p. 17
Abstract
from arXiv · showhide
Scaling laws hold that language models grow more capable with more parameters and more training data. Mixture-of-Experts (MoE) architectures are a remarkable demonstration of these laws, activating only a fraction of an enormous parameter bank for each token. But this success is built on static pretraining data --- the facts and corrections supplied by users during live interactions are a significant untapped source of potential improvement for a deployed model, but cannot be exploited by conventional architectures whose weights are frozen after training. Instead, this newfound knowledge must be placed in the context (by instruction or retrieval) and re-read on every request, only to be discarded afterwards. We seek instead to learn from live interactions by dynamically updating model weights. Inspired by MoEs, we propose the \textbf{Infinite-Parameter LLM}. A compact hypernetwork turns the online data into low-rank modulations of a shared base network, so feed-forward weights are generated from live data, not read from static memory. Whereas existing weight generators are held fixed after reading the context once, we form a Bayesian belief over the generator's latent state and update it online, such that the effective weights are re-derived as our belief evolves during the session. Although the model's memory footprint is constant, the feasible space of generated weights is thus effectively infinite. Representing live data in the weights rather than the prompt amortises compute, frees the context window, persists updates across turns, and can generalise better than in-context use. Our evaluation protocol applies this methodology to in-context learning and retrieval.
1 Introduction
Live interactions provide a growing source of facts, corrections, and outcomes, but conventional deployed models keep their weights frozen and carry new information in prompts. The paper proposes generating low-rank weight updates from live data and adapting them online through a belief over the generator’s latent code.
1 Introduction
- Live interaction logs offer facts, corrections, and outcomes that can help models become more useful on particular users’ tasks.The paper presents these logs as a growing data source as public human-generated text becomes finite.Source: §1 · p. 1
- Frozen models use prompting or retrieval to carry new information in context, where it is reread for each request and then discarded.The paper argues that genuine test-time learning requires weights that absorb new information without forgetting existing knowledge.Source: §1 · p. 1
- The generated-weight approach has a fixed memory footprint but an unbounded set of reachable effective weights and behaviours, not unbounded knowledge.Adaptations are described as bounded, low-dimensional, and reversible, and as complementary to pretraining.Source: §1 · p. 2
- A compact generator produces low-rank updates to a shared base network, while a belief over its latent code changes online as the interaction proceeds.The belief can be updated contextually, per turn, or per token.Source: §1 · p. 3
- The paper frames its evaluation as a comparison between carrying information in weights and carrying it in prompts, including in-context learning and retrieval baselines.Other listed baselines include one-shot weight generators, point-estimate test-time training, and vanilla MoE approaches.Source: §1 · p. 3
2 Related Work
Related work covers expert selection and compression, generated weights, inference-time adaptation, continual learning, and prompt-based conditioning. The paper positions its contribution as coupling weights generated from live data with an online-updated belief over the generating code.
2.1 Mixture-of-Experts: from stored banks to bank-free selection
- Stored-bank MoEs select among fixed experts, while bank-free approaches avoid materializing experts but still rely on fixed factors or routing; this work generates an additive low-rank update and adapts its code online.The paper distinguishes its shared-base FFN from stored expert banks and frozen bank-free selection.Source: §2.1 · p. 3
2.2 Compressing and factorising experts
- Expert compression reduces the cost of stored banks, whereas this approach removes the bank and generates experts from a latent code produced from live data.The comparison with µMoE emphasizes that its factorized tensor is fixed and not adaptable.Source: §2.2 · pp. 3–4
2.3 Hypernetworks and generated experts
- Context-driven weight generators encode data into adapters, but read context once and freeze the result; this work adds a belief over the code that evolves during interaction.Prior generators span compact task-description encoders and more resource-intensive context encoders.Source: §2.3 · p. 4
2.4 Inference-time adaptation and fast weights
- Inference-time adaptation methods update weights or routing, but the cited LLM approaches use point estimates or fixed expert banks; this work filters a generated FFN code while leaving attention unchanged.Its update is an amortized approximation to an explicit Bayesian filter.Source: §2.4 · p. 4
2.5 Continual, online, and Bayesian foundations for adaptation
- Continual-learning methods address stability and plasticity; this work applies uncertainty-gated adaptation to a low-dimensional generated code, making changes bounded and reversible.The related Bayesian work includes filtering and amortized inference, while this method updates online at inference.Source: §2.5 · p. 5
2.6 Conditioning on run-time data through the prompt
- Prompt conditioning and retrieval place run-time information in context for each request, whereas this paper measures a weight-carried alternative against those approaches.Context-carried information competes for a bounded window and is discarded when the request ends.Source: §2.6 · p. 5
2.7 Positioning: how this work differs
- The paper claims novelty in coupling live-data-driven weight generation with an online-updated belief, not in generation instead of storage alone.Its closest context-driven generator relatives freeze the adapter, while inference-time adaptation relatives do not generate weights from live data and update a belief over the generating code.Source: §2.7 · pp. 5–6
3 Method: Generating and Adapting FFN Experts
The method generates low-rank FFN modulations from live data and adapts them by updating a belief over the latent code. It keeps resident parameters fixed while making an unbounded set of effective weights reachable.
3.1 Overview
- Selected FFNs combine an always-applied shared base with a generated low-rank modulation, while a belief over its latent code evolves from live data.The expert for each token is generated and discarded rather than stored in a bank.Source: §3.1 · pp. 6–7
3.2 The belief-over-code framework
- A distribution over the latent code is the framework’s primitive: live data writes it, and its summary drives the generated weight delta.The delta is an additive low-rank change to the shared base; the framework permits Gaussian or categorical beliefs.Source: §3.2 · pp. 7–9
- Encoder designs trade footprint against reading fidelity, from a small readout head to a dedicated context encoder or separate hypernetwork.Online belief updates can correct an imperfect one-shot read; attention and base weights remain frozen.Source: §3.2 · p. 9, §3.3 · p. 9
3.3 Moving the belief: online inference over the code
- A, B, and C update the code belief contextually, per turn, or per token; explicit variants preserve an FFN-side belief, with C using an amortized filter.The posterior’s spread can calibrate trust, gate plasticity, and support controlled forgetting, but calibration under amortization remains to be validated.Source: §3.3 · pp. 9–10
3.4 The infinite-parameter view
- The reachable effective weights are unbounded although stored parameters are finite, because codes are materialized from data rather than selected from a fixed bank.A categorical belief is a finite working set over this generated space; “infinite” concerns reachable configurations, not stored knowledge.Source: §3.4 · pp. 10–11, Figure 4 · p. 11
3.5 Architecture choice in this paper
- The evaluated architecture uses a categorical belief over data-materialized codes; a continuous-Gaussian belief is described but not evaluated.The categorical posterior selects a top-1 code or its mean and updates as evidence accumulates.Source: §3.5 · p. 11, §3.5.1 · p. 11, §3.5.2 · p. 12
3.5.1 A categorical belief over materialised codes
- The categorical selector scores each code against layer activations and applies one selected expert, updating its belief per turn or per token.The measurement channel supplies codes, while the selector performs per-step inference over the pool.Source: §3.5.1 · p. 11
3.5.2 The low-rank generator
- The generator uses a frozen SwiGLU base and SHINE’s memory-token/M2P pipeline to produce rank-8 FFN deltas from 128-dimensional codes.Codes are cached per knowledge set, so generation need not reread evidence tokens; the paper adds a small categorical selector to the frozen pipeline.Source: §3.5.2 · p. 12
4 Experiments
Experiments test whether evidence can be compiled into weights, routed among codes, and accumulated across turns. Results favor weights over prompts for long, noisy evidence, trained selection over dense retrieval, and persistent belief for context-dependent routing.
4.2 Data-to-weights beats the prompt where evidence is long and multi-hop
- On MS MARCO, data-to-weights reaches 48.0 F1 vs in-context 33.6, while SQuAD favors in-context 85.3 vs data-to-weights 51.8.The comparison reverses as evidence moves from one short, clean passage to longer, noisier evidence.Source: Table 4 · p. 13, §4.2 · p. 13
4.1 Setup
- The evaluation covers five QA datasets spanning evidence length, noise, and multi-hop structure, using answer F1 or top-1/top-3 routing accuracy over 150 held-out groups.SQuAD is used only for weights-versus-prompt comparison; multi-passage sets provide relevance labels for routing.Source: §4.1 · p. 13
4.2 Data-to-weights beats the prompt where evidence is long and multi-hop
- At a 3000-token budget with no truncation, oracle F1 falls from 51.6 to 46.9 as distractors grow from 8 to 32, showing fixed-code saturation.At 1300 tokens, all reads truncate by 32 passages and realistic F1 collapses to 27.8.Source: Figure 5 · p. 14, §4.2 · p. 14
4.3 A trained belief over the code pool beats retrieval
- The trained selector beats dense retrieval on every dataset, by 8 F1 on MS MARCO (53.3 vs 45.3) and 10–12 on multi-hop sets.The selector scores code keys against a query derived from the base model’s activations.Source: §4.3 · p. 14, Table 5 · p. 15
- At 64 passages, the single big read falls to 27.8 F1 while selection over per-passage codes stays flat as the pool grows.Each code is compiled from one bounded passage, avoiding a single over-length read.Source: Table 6 · p. 15, §4.3 · p. 15
4.4 Cross-turn accumulation: the belief sharpens as the conversation grows
- Accumulated belief routing rises on context-dependent turns while memoryless routers stay flat and concat-history retrieval eventually sags; belief cost stays flat per turn.The comparison uses conversations with a fixed knowledge pool and reports only context-dependent turns.Source: §4.4 · pp. 15–16, Figure 6 · p. 16
5 Efficiency analysis of live weight adaptation vs fine-tuning
Live weight adaptation is presented as substantially cheaper than fine-tuning in compute and storage, with advantages that grow at scale. These savings are amortised over a one-time generator investment.
5 Efficiency analysis of live weight adaptation vs fine-tuning
- 3S is the marginal compute ratio of fine-tuning to data-to-weights compilation, independent of base-model size P and data length D.The accounting compares S fine-tuning steps at 6PD each with one 2PD encoder pass; the table considers S from 100 to 1000.Source: §5 · p. 16, Table 7 · p. 17
- Over 1000× per-item storage reduction is possible by quantising the cached code to fp8 while keeping the base model at fp16.The comparison is against a full-parameter checkpoint; the table specifies a Qwen3-8B-scale base and one 2048-token data point.Source: §5 · p. 16, Table 7 · p. 17
- Fine-tuning can add multi-GPU sharding, communication, and redeployment overhead, while forward-pass weight adaptation avoids those costs.The comparison describes deployment beyond a single card.Source: §5 · pp. 16–17
- The reported savings are amortised over a one-time generator-training investment, which is outside the paper’s scope.The authors characterize compilation as cheaper than fine-tuning by orders of magnitude in compute, with storage advantage widening as models scale.Source: §5 · p. 16
6 Limitations
The design’s scope is limited by what its compact generator can represent and by the supplied data and adaptation setting. Its demonstrated advantages are not universal across prompts, data pools, or conversation conditions.
6 Limitations
- A compact generator cannot carry a large MoE’s stored knowledge because knowledge remains bounded by parameter count.Without relevant facts or instructions supplied at run time, the model retains only its base knowledge and default behaviour.Source: §6 · p. 17
- Weight compilation is advantageous over prompting specifically for large or repeatedly reused data and long horizons; short, one-use context is a strong, cheap baseline.The comparison is weights versus prompt, not closed-book performance with no supplied information.Source: §6 · p. 17
- Per-token generation adds bandwidth cost, requiring a small generator and low-rank deltas.Source: §6 · p. 17
- The evaluated belief is categorical over materialised codes and selects top-1; a shift outside the pool requires materialising a new code.Continuous-Gaussian belief is left for future work, and top-1 selection can collapse a multimodal posterior.Source: §6 · p. 17
- Belief accumulation is demonstrated on authored multi-turn conversations and pools they were built from, leaving longer horizons and natural sessions for stress-testing.The forgetting control γ also remains a condition to test further.Source: §6 · p. 17
7 Conclusion
The paper proposes generating experts from live data over a shared base and updating a belief over their code during interaction. Experiments report benefits over prompting on long, noisy evidence and improved code routing as conversations grow.
7 Conclusion
- Live-data-generated experts use a shared base and an online-updated belief over the generating code, rather than a fixed bank of stored experts.The implemented framework uses generated low-rank additive deltas and a categorical belief over materialised codes, selected top-1 and sharpened online.Source: §7 · pp. 17–18
- On long, noisy evidence, compiled run-time data beats the prompt; trained belief identifies the right code at least as well as strong retrieval.Source: §7 · p. 18
- As a session grows, belief accumulated across a conversation routes better than a single read or rereading the growing history.Source: §7 · p. 18
- “Infinite” means unbounded reachable effective weights and behaviours from live data within a fixed memory footprint.Source: §7 · p. 18