Source-linked AI summary
Internalizer: Portable Context-to-Parameter Mapping for Very Large Language Models
Peter Devine, Nick Ryan, Benjamin Sirb, Alex Chiocchi
TL;DR
Some LLM uses remain too costly or insufficiently accurate to justify deployment. The Internalizer maps document context into a LoRA adapter through a portable hypernetwork. On unseen documents up to 4096 tokens, it reaches 84.9% top-1 and 97.8% top-5 teacher-forced accuracy.
Problem
For some use cases, LLM inference remains too expensive or model accuracy too low to justify deployment.
Source: §1 · p. 1Method
Results
On unseen documents up to 4096 tokens, adapters reach 84.9% top-1 and 97.8% top-5 teacher-forced accuracy, roughly 22 points of top-1 over the base model.
Source: §8 · p. 12Takeaways & Limitations
After training, one forward pass can turn a document into an adapter to serve alone for speed or alongside the context for accuracy.
Source: §1 · p. 2The authors present the hypernetwork as a pre-trained basis, with real-task fine-tuning and deployment accuracy evaluation still needed.
Source: §8 · p. 13
Abstract
from arXiv · showhide
Hypernetworks that map a context directly to a LoRA adapter let a large language model carry that context in its weights, but prior work has demonstrated them only on base models of up to 14 billion parameters. We present the Internalizer, a state-of-the-art, portable Context-to-Parameter Mapping hypernetwork that generates document-specific LoRA adapters for the frozen 284B-parameter DeepSeek v4 Flash, a target two orders of magnitude larger than in any previous work. Most of its parameters live in a model-agnostic trunk with only thin entry and exit layers per base model, so it trains cheaply against small models before being ported to the large one. On unseen documents of up to 4096 tokens, the generated adapters reach 84.9% top-1 and 97.8% top-5 teacher-forced accuracy against 63.4% and 83.5% for the base model, with nothing in the context window but a three-word instruction. Once the hypernetwork is trained, a single forward pass turns any document into an adapter for such a model, which could be served alone for speed or alongside the document in the window to raise accuracy further.
1 Introduction
The introduction frames context-specific LoRA generation as a response to deployment cost and accuracy challenges, then presents a portable hypernetwork for very large LLMs. It reports document internalization on a 284B target and describes serving the generated adapter with or without the source context.
1 Introduction
- LLM deployment can be limited by inference cost or low accuracy; long contexts raise latency and can erode accuracy, complex in-context prompts add cost, and task fine-tuning is slow and input-agnostic.The introduction also identifies bloated context windows, inefficient in-context learning, and impractical per-task training as challenges.Source: §1 · p. 1
- Context-to-Parameter Mapping converts tokenizable context into a LoRA adapter, then uses an ordinary user question, command, or instruction to produce output.The context may be a document, code file, table schema, image, or other tokenizable input.Source: §1 · p. 1
- A context-specific adapter can be computed once and cached, then served instead of the context for speed or alongside it in accuracy mode.The paper presents the latter as the mode expected to matter most in practice.Source: §1 · p. 1
- The model-agnostic hypernetwork is intended to reduce retraining costs across LLMs, while a single forward pass can generate adapters for documents, schemas, or code files on demand.The paper also describes using an adapter alone for speed or with its source context for accuracy.Source: §1 · p. 2
- Oversized entry and exit projections, shared gated-MLP adapter slots, an error-gated hard-token loss, and reverse distillation on small models support portability and scaling.The trunk’s representational capacity is decoupled from target hidden width, while most learning occurs against smaller base models before porting.Source: §1 · p. 2
- 84.9% top-1 teacher-forced accuracy versus 63.4% for the base model on documents up to 4096 tokens, using a 284B-parameter target twenty times larger than prior work.The target is DeepSeek v4 Flash 0731.Source: §1 · p. 2
2 Related work
Prior context-internalizing systems showed promise but were tied to individual base models and small targets. The related work contrasts those limits with portability efforts and motivates the Internalizer’s shared-core design and small-to-large training strategy.
2 Related work
- GenerativeAdapter (Chen et al., 2025) accumulates hidden states into LoRA, while StreamAdapter (Muhtar et al., 2024) compresses evicted KV cache and reached a 14B target; both are base-model-specific.Both retain fixed adapter capacity as context grows, unlike the Internalizer’s context-proportional rank.Source: §2 · p. 3
- Doc-to-LoRA (D2L) uses chunked Perceiver latents and growing adapter rank, but its self-distillation, document-QA data, retraining needs, and 2–7B evaluations constrain it.The Internalizer adopts D2L’s chunked Perceiver design as its starting point.Source: §2 · p. 3
- LatentSkill (Yu et al., 2026) internalizes SKILL.md and outperforms in-context use but targets one task and backbone; SADA links fidelity to supervision quality and accuracy to rank, yet remains base-specific.SADA evaluates only 1.5–7B backbones.Source: §2 · p. 3
- PorTAL validates a shared core with thin per-model maps, recovering ∼98% of a from-scratch LoRA’s lift, but uses task latents without context input and targets 1.7–8B bases.Each new task requires its own training run rather than a forward pass.Source: §2 · p. 3
- Reverse distillation transfers hypernetwork learning from small base models to a much larger target, inverting the conventional large-teacher-to-small-student direction (Hinton et al., 2015).The authors say this strategy is novel for context-to-parameter mapping.Source: §2 · p. 3
- The Internalizer addresses base-model binding and the prior ∼14B target ceiling with a shared trunk, thin per-model entry and exit layers, and a 284B target.The paper describes the model-specific layers as ≲10% of parameters.Source: §2 · p. 3
3 Architecture
The Internalizer turns activations from a frozen base model into a context-specific LoRA adapter while keeping most computation in a portable, shared trunk. Oversized entry and exit projections, chunk-wise encoding, and MLP-only adapter slots support reuse across models.
3 Architecture
- The Internalizer converts frozen base-model activations into a LoRA adapter, with most computation in shared weights and only thin input and output layers specific to each model.Its architecture has six parts, including a perceiver, bias queries, a shared decode head, and a combine step.Source: §3 · p. 4, §3.7 · p. 6
3.1 A hidden state input adapter
- Model-specific projections expand activations into an 8192-wide entry space before shared weights compress them to the 1536-wide trunk latent space.The oversized intermediate space avoids forcing the model-specific projection to discard information and decouples trunk capacity from base-model hidden width.Source: §3.1 · p. 4
3.2 A depth embedding
- Relative layer depth lets models with different layer counts use the same smooth shared embedding, while a model-specific table absorbs absolute layer differences.The relative depth runs from 0 to 1; the shared component uses Fourier features at eight frequencies.Source: §3.2 · p. 4
3.3 A perceiver
- A perceiver compresses each layer’s chunk of variable-length tokens into 256 latent vectors, then decodes them into vectors for the layer’s module and rank slots.Each layer-chunk group is processed independently, enabling chunks to be processed in parallel and recombined later.Source: §3.3 · p. 5
3.4 Bias queries
- A separate bias-query stream generates a context-independent adapter component once, avoiding duplication across a document’s chunks.The bias latents use learned queries and depth signals, without token inputs or attention.Source: §3.4 · p. 5
3.5 A shared decode head
- The shared decode head converts chunk and bias latents into LoRA weight rows, using a large shared intermediate and thin model-specific projections to the target weight shapes.Zero-initialized B-side scalers make the generated update initially zero, so the base model starts unmodified.Source: §3.5 · p. 5
3.6 A combine step
- Concatenating the independently encoded chunk blocks and one bias block yields a standard LoRA adapter of rank r(k+1), where k is the number of chunks.Each added chunk contributes its own parameters, and the fixed scaling factor avoids shrinking the update as chunks are added.Source: §3.6 · pp. 5–6
3.7 Targeted modules
- The Internalizer targets the up, gate, and down projections in each layer, restricting slots to the shared gated-MLP structure so one trunk can port across varied backbones.For DeepSeek v4 Flash, the slots are placed on each layer’s shared-expert MLP; no attention modules are targeted.Source: §3.7 · p. 6
4 Pre-training
Pre-training teaches the hypernetwork to encode document content into adapters through document regurgitation. An error-gated objective emphasizes unlearned tokens, while a small-model warm start precedes porting to DeepSeek v4 Flash.
4 Pre-training
- Pre-training first teaches the hypernetwork to internalize as much of each context as possible, deferring task-specific fine-tuning to later work.The curriculum uses fresh corpus slices and progressively raises its document-length cap, with twelve small-model stages followed by four target-model stages.Source: §4 · p. 6
4.1 The regurgitation task
- The regurgitation task asks the model to reproduce a document from its generated adapter using the fixed instruction “Reproduce the document.”The instruction carries no document-specific information, and the fixed input is replaced by real task inputs at fine-tuning time.Source: §4.1 · p. 6
4.2 Data
- Training documents are filtered English FinePDFs split into contiguous chunks of at most 512 tokens, each tagged with its document identifier and chunk position.Training stages use fresh corpus shards, and chunk tags remain in place at evaluation and inference.Source: §4.2 · p. 7
4.3 Training loop
- The frozen base model encodes tagged chunks, the hypernetwork generates rank-16 LoRA blocks, and teacher-forced next-token loss updates only hypernetwork weights.Loss is computed over the assistant document span; the instruction and chat scaffolding are masked out.Source: §4.3 · p. 7
4.4 Error-gated loss
- An error-gated term gives extra gradient weight to tokens the modulated model predicts incorrectly, while plain cross-entropy remains as an anchor.The term is introduced partway through pre-training, with λ ranging from 0.2 in early stages to 1.5 in the final DeepSeek v4 Flash stages.Source: §4.4 · p. 7
4.5 Warm-starting on small models
- Reverse distillation trains the shared trunk against five 1–3B-parameter models before porting it to the larger target, reducing training cost.The small-model curriculum raises document caps from 64 to 4096 tokens; its stage boundaries were chosen ad hoc, while the warm start is identified as the key benefit.Source: §4.5 · pp. 7–8
4.6 Porting to DeepSeek v4 Flash
- The warmed hypernetwork is ported to 284B-parameter DeepSeek v4 Flash through stages that first train model-specific layers, then unfreeze the full system.The porting phase uses the same data recipe and progresses from 512-token single-chunk documents to 4096-token chunked documents.Source: §4.6 · p. 8
5 Evaluation Method
The evaluation measures how much document content a generated adapter lets DeepSeek v4 Flash reproduce without the document in its input. It compares adapter-only regurgitation with the unmodified model across document sets and context lengths.
5 Evaluation Method
- The final Internalizer generates an adapter for each document and length, and the modulated model reproduces the document from that adapter alone.The unmodified base model’s language modeling of the same tokens provides the comparison.Source: §5 · p. 8
5.1 Data
- Evaluation uses FinePDFs documents from the training distribution, long FinePDFs documents, and PG-19 books, including out-of-domain narrative prose.Documents are evaluated at seven length buckets from 256 to 16384 tokens; longer buckets contain fewer eligible documents.Source: §5.1 · p. 8
5.2 Adapter generation
- Adapters are generated by truncating, chunking, tagging, and encoding each document, then combining rank-16 chunk blocks with a bias block.For k chunks, this produces a standard LoRA adapter of rank 16(k+1).Source: §5.2 · p. 8
5.3 Metrics and control
- Perplexity, top-1 accuracy, and top-5 accuracy measure teacher-forced document reconstruction, with the unmodified base model scored on raw document tokens as control.Top-5 accuracy credits cases where the true token is among the five highest-probability predictions; the table compares adapter-only results with the base model by document set and context length.Source: §5.3 · pp. 8–9, Table 1 · p. 9
6 Results
The Internalizer reconstructs unseen documents effectively within its trained range, but accuracy degrades beyond that range. A portability experiment finds that a warmed shared trunk transfers to the 284B target, with thin model-specific layers accounting for much of the porting work.
6 Results
- The adapter and base-model evaluations are not perfectly matched: the adapter setting also supervises chunk tags and turn close, about 1.5% more tokens.This qualifies the comparison between the two supervised spans.Source: §6 · p. 9
6.1 Within the trained length
- 84.9% top-1 and 97.8% top-5 accuracy on documents up to 4096 tokens compare with 63.4% and 83.5% for the base model.Macro-averaged across five length buckets and three document sets, adapter perplexity is 1.7 against 5.6 for the base model; evaluation uses teacher forcing with only a three-word instruction in the context window.Source: §6.1 · p. 9
- Base-model accuracy stays largely flat across lengths, while adapter accuracy varies, so their gap reflects how much document content reached the adapter rather than text predictability.The first clear accuracy drop occurs at 4096 tokens, where the chunk count reaches the training cap; adapter rank grows with chunk count and capacity scales with content.Source: §6.1 · pp. 9–10
- PG-19 reaches 81.0% macro top-1 through 4096 tokens, versus 86.2% and 87.5% for the FinePDFs sets.The passage attributes the gap to internalization: the base model scores slightly higher on PG-19, whose narrative prose is out of domain and lacks extracted-PDF layout regularity.Source: §6.1 · p. 10
6.2 Beyond the trained length
- At 8192 tokens, top-1 margins over the base model are 0.3 to 3.1 points; at 16384 tokens, adapter accuracy falls to 22–25% top-1 with perplexities in the thousands.These extrapolation lengths require 16 and 32 chunks, respectively, beyond the eight-chunk training maximum; the 16384-token bucket is omitted from Figure 2 because adapter perplexities would dominate its axis.Source: §6.2 · p. 10, Figure 2 · p. 10
6.3 Portability
- Both warm-started porting runs pass 90% top-1 within the training budget, while cold start remains below 75% at the same step count.The frozen-trunk run gains accuracy in fewer steps, and the fully unfrozen run finishes marginally ahead.Source: §6.3 · p. 11
- A trunk trained against 1–3B models can be fitted to the 284B target through thin entry and exit layers, supporting reverse distillation as the cheap bulk-learning strategy.The expensive large-model training then needs to fit only the thin model-specific layers.Source: §6.3 · p. 11
7 Future work
The paper evaluates the Internalizer on document reconstruction and outlines follow-up directions. These include adapting it to tasks and harnesses, extending context length and model scale, and exploring additional adapter targets and reflection-based refinement.
Task finetuning
- Task fine-tuning would replace the reconstruction prompt with real questions, instructions, and answers for applications such as document QA, code generation, and grounded instruction following.Each task fine-tune would branch from the same pre-trained hypernetwork.Source: §7 · p. 11
Full-context finetuning
- Full-context fine-tuning would keep the context both in the window and as hypernetwork input, so the adapter specializes to the task and context subtype.The proposed setup combines task generalization with specialization to each context.Source: §7 · p. 11
Longer context lengths
- Longer useful context ranges must be trained for rather than expected through extrapolation; the authors plan to raise the cap beyond eight chunks and measure accuracy and cost.Source: §7 · pp. 11–12
Transfer to other and larger models
- The authors expect a trunk trained on DeepSeek v4 Flash to transfer more cheaply to other large models than training from scratch, and identify trillion-parameter models as targets.This is an expectation, with Qwen3.8-2.4T-A95B and Kimi K3 named as examples.Source: §7 · p. 12
Harness-specific training
- Harness-specific fine-tuning could let adapters absorb deployed tools, prompting conventions, and output formats.The authors describe this as the route they expect commercial applications to take.Source: §7 · p. 12
Self-improvement of adapters through reflection
- A strong but slow model could review adapter mistakes or user interactions, add learned information to the context, and regenerate an improved adapter offline without weight training.Source: §7 · p. 12
N-gram embedding targets
- Adapter components targeting n-gram embedding tables, or task-relevant subsets, could pair internalization with embedding offloading for further inference savings.The proposal references the Qwen 3.8 Next architecture.Source: §7 · p. 12
8 Conclusion
The Internalizer generates portable document-specific LoRA adapters for a frozen 284B model, with strong document-reproduction accuracy and two serving modes. The work presents this as a pre-trained basis, with real-task finetuning and deployment evaluation still to come.
8 Conclusion
- The portable hypernetwork targets frozen DeepSeek v4 Flash at 284B parameters, twenty times larger than any prior target in this area.Its trunk is shared across base models, with only thin model-specific entry and exit layers.Source: §8 · p. 12
- 84.9% top-1 and 97.8% top-5 teacher-forced accuracy on unseen documents up to 4096 tokens, with nothing in the context window.The result is reported for the pre-trained Internalizer.Source: §8 · p. 12
- The generated adapter carries task and context information that a standard LoRA adapter cannot.Source: §8 · p. 13
- Adapters can be swapped in per request and reused across documents, schemas, or codebases; trunk portability lets them follow base-model upgrades after refitting the thin entry and exit layers.Source: §8 · p. 13
- The work is a pre-trained basis, not a finished model; real-task finetuning and measurement of deployment accuracy gains remain future work.Source: §8 · p. 13
Contributions
The passage credits the project’s research, consultation, and supervision contributions to four named contributors.
Contributions
- Peter Devine led the research, co-created the concept, designed and ran experiments and evaluations, and wrote the paper.Source: Contributions · p. 13
- Nick Ryan and Ben Sirb consulted, contributed research-direction input, and reviewed drafts.Source: Contributions · p. 13
- Alex Chiocchi supervised the work, co-created the concept, consulted on its direction, and reviewed the paper.Source: Contributions · p. 13