PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Laws for Hypernetwork-Based Knowledge Injection in Large Language Models

Nischay Dhankhar, Dos Baha, Abulhair Saparov

arXiv:2607.19604v1cs.CLcs.LG

TL;DR

Reliable, scalable factual knowledge injection remains difficult, and hypernetwork scaling lacks systematic characterization. This paper studies train-time hypernetwork injection across architectural and target-model scales, finding predictable power-law improvements and stronger OOD scaling than LoRA and full fine-tuning.

  • Problem

    Factual knowledge injection remains difficult because existing fine-tuning methods can be expensive, prone to catastrophic forgetting, and weak in OOD generalization, while hypernetwork scaling lacks systematic characterization.

  • Method

    The paper trains hypernetworks to generate fixed LoRA adapters for frozen language models and evaluates scaling across hypernetwork architecture, target-model size, injected-fact count, and multi-hop MegaWikiQA tasks.

  • Results

    Hypernetwork injection shows smooth, predictable scaling, with steeper OOD generalization scaling than LoRA and full fine-tuning while achieving comparable in-distribution performance.

  • Takeaways & Limitations

    Target-model scaling yields the strongest gains, while hypernetworks generalize across unseen domains, entities, relations, rephrased queries, and formats.

  • Takeaways & Limitations

    At the largest tested scale, the hypernetwork reached approximately 2.5B parameters for a 1.5B target model, limiting practical deployability.

Abstract

from arXiv · show

Injecting factual knowledge into large language models (LLMs) reliably and at scale remains an open challenge. Hypernetworks provide a promising solution to large-scale knowledge injection. Although hypernetworks are typically applied for test-time adaptation, we explore their use in train-time knowledge injection, where, given a large corpus of facts, we train a hypernetwork to generate a fixed LoRA adapter that, when inserted into the target model, enable the model to answer questions about those facts. In this work, we investigate whether hypernetworks can be used to perform train-time knowledge injection and how this ability varies with scale. The scaling behavior of hypernetworks remains largely unstudied. Our design decouples the hypernetwork's injection capacity from the target model's general capability, enabling, for the first time, a rigorous study of scaling laws for hypernetwork architectures. We characterize how loss, reasoning accuracy, and out-of-distribution (OOD) generalization vary with hypernetwork depth, width, and target network size. We construct a large-scale dataset, called MegaWikiQA, containing tens of millions of multi-hop question-answer examples across 39 domains constructed from examples in Wikidata5M. Our results reveal: (i) hypernetwork-based injection exhibits broadly predictive power law scaling along all architecture axes; and (ii) hypernetworks are capable of reliable OOD generalization at increasing scales, suggesting that hypernetwork provides a promising alternative to other train-time adaptation methods such as LoRA finetuning and full fine-tuning, exhibiting steeper scaling exponents in all OOD evaluations. Together, these results establish hypernetworks as a principled and scalable substrate for train-time adaptation, and provide the first empirically grounded scaling laws to guide hypernetworks for factual reasoning in large language models.

1 Introduction

The paper studies train-time factual knowledge injection with hypernetworks, which generate contextual adaptations for frozen target models to address fine-tuning costs, forgetting, and weak OOD generalization. It provides the first systematic scaling-law analysis across hypernetwork architecture, target-model size, and injected-fact quantity, finding predictable scaling and stronger OOD scaling than fine-tuning baselines.

  • Motivation: Full fine-tuning is prohibitively expensive, while LoRA and other fine-tuning methods remain vulnerable to catastrophic forgetting and poor OOD generalization.These limitations motivate an alternative knowledge-injection mechanism for knowledge-intensive applications.
  • Method: Hypernetworks condition on injected facts to generate contextual adaptations such as LoRA adapters while leaving the target model’s base parameters unchanged.The approach is intended to mitigate the limitations associated with modifying target-model parameters.
  • Method: Train-time knowledge internalization compresses a fixed fact corpus into target-model weights, with evaluation queries having no inference-time access to that corpus.Both the hypernetwork and the compared fine-tuning baselines are trained under this setup.
  • Scaling study: The study is the first systematic characterization of scaling laws across hypernetwork depth and width, target language-model size, and injected-fact quantity.It also examines regimes where scaling yields diminishing returns and compares depth and width scaling.
  • Results: Hypernetwork-based injection exhibits smooth, predictable scaling, generalizes across unseen entities and relations, and supports multi-hop compositional queries.These results contrast with sharp failure modes reported for parametric editing.
  • Results: As target-model size increases, hypernetwork-generated adaptations show steeper OOD generalization scaling than LoRA and full fine-tuning while achieving comparable performance on in-distribution evaluations.The introduction presents this as evidence that hypernetworks are a promising scalable alternative for train-time adaptation.

2 Related Work

Prior work frames knowledge injection as teaching pretrained models to condition outputs on a specified text corpus, using fine-tuning, parameter-efficient adaptation, and hypernetworks. Existing scaling and dataset studies leave open the need to analyze hypernetwork architecture scaling and large-scale evaluation for knowledge injection.

  • Fine-tuning and PEFT for Knowledge Injection: Fine-tuning and parameter-efficient methods have been widely used to incorporate corpus knowledge into pretrained language models.Early examples include K-BERT, K-Adapter, and ERNIE.
  • Hypernetworks for Knowledge Injection: Hypernetworks generate another network’s weights and have supported parameter-efficient adaptation across tasks and languages, including zero-shot transfer to unseen task-language combinations.Hyper-X conditions adapter weights on joint task and language embeddings.
  • Knowledge injection: Knowledge injection denotes teaching a pretrained model a body of knowledge from a text corpus, even when facts may overlap with pretraining.The goal is to reliably condition model outputs on the specified corpus.
  • Scaling Laws for Hypernetworks: Prior hypernetwork scaling work varies target network size for image and audio reconstruction but does not scale hypernetwork architecture or study knowledge injection.Generating weights for a separate frozen target network from structured inputs raises distinct scaling questions.
  • Knowledge Injection Datasets: Existing knowledge-injection benchmarks range from tens of thousands of examples in ZsRE to 311K in UniEdit and 500K in WikiBigEdit.These benchmarks differ substantially in scale and construction methodology, while ZsRE predates modern LLMs.

3 Dataset Construction: MegaWikiQA

MegaWikiQA is designed for scalable, reliable evaluation of factual knowledge injection by combining deterministic, structurally rich question generation with single- and multi-hop reasoning over Wikidata5M. The dataset uses random graph walks, grammar-based verbalization, unambiguous relations, domain balancing, and separate ID/OOD evaluation sets.

  • Dataset goals: MegaWikiQA supports scaling-law studies by requiring large scale, structurally rich examples with deterministic labels for single-hop and multi-hop compositional reasoning.Multi-hop evaluation tests reasoning over multiple injected facts rather than isolated fact recall.
  • Question generation: A recursive grammar converts each walk into a noun phrase and question, while a manually curated mapping selects templates for all 822 Wikidata relations.The procedure is deterministic and avoids neural generation during question construction.
  • Label determinacy: Restricting the dataset to one-to-one and many-to-one relations reduces label uncertainty and enables more reliable accuracy evaluation.One-to-many and many-to-many relations are excluded because their answers are inherently non-deterministic.
  • Dataset scale: The full generation process produces approximately 10 million examples per hop count up to 4 hops, classified across 39 domains and filtered for classification confidence and hop-count balance.The resulting dataset is built from the full Wikidata5M corpus.
  • Evaluation sets: Evaluation uses disjoint ID and OOD sets, including 10,000 stratified ID examples and 10,000 OOD examples from philosophy, linguistics, and civil engineering with GPT 4.1-rephrased questions.ID examples are disjoint from training at the triplet level, while OOD domains were selected for diversity and distinctiveness.

4 Method

The method trains a randomly initialized transformer hypernetwork to identify relevant facts among distractors and generate weight adaptations for a frozen language model. It systematically varies hypernetwork depth, width, and injected-fact count while holding other variables constant.

  • Knowledge injection formulation: Each example pairs a query with N facts, exactly one relevant and N−1 uniformly sampled negatives, requiring relevant-fact identification.Unless otherwise stated, experiments use N = 4 facts per example.
  • Knowledge injection formulation: The hypernetwork encodes the relevant fact into weight adaptations that steer the frozen target model toward the correct answer.This formulation separates fact selection from adaptation of the fixed model.
  • Hypernetwork design: The transformer hypernetwork is initialized entirely from random weights, preventing pretrained initialization from confounding architectural-capacity effects.The target language model remains frozen with fixed parameters.
  • Architecture scaling axes: Scaling experiments independently vary depth across {1, 2, 4, 8, 16}, width across {32, 64, 128, 256, 512, 1024}, and fact count N.All other variables remain constant when one scaling variable changes.

5 Experiments

Across hypernetwork width, depth, target-model size, and fact-count scaling, evaluation losses follow broadly smooth power laws, with target-model scaling producing the steepest exponents and OOD rephrased scaling remaining consistently flattest. Compared with LoRA and full fine-tuning, hypernetworks scale slightly worse on validation loss but best on all three OOD metrics, with their advantage widening as target models grow.

  • Evaluation setup: All four evaluation metrics are assessed through power-law fits: ID validation, OOD non-rephrased, OOD rephrased, and OOD MCQ loss.Each OOD split tests a qualitatively distinct form of generalization.
  • Hypernetwork scaling: Width and depth both produce smooth power-law scaling, with comparable validation exponents of −0.096 and −0.088, respectively.Width scaling benefits ID validation and OOD non-rephrased similarly, while rephrased OOD improves more slowly; depth shows the same pattern.
  • Target-model scaling: Target-model scaling yields the steepest exponents: −0.226 for validation loss and −0.184 for OOD non-rephrased loss, versus −0.088 and −0.096 for depth and width validation scaling.The 0.5B target model lies above the fit, suggesting a minimum target capacity threshold for effectively leveraging generated LoRA adaptations.
  • Fact-count scaling: Fact-count scaling improves every metric, but modestly, with exponents of −0.080 for validation loss, −0.077 for OOD non-rephrased, and −0.028 for OOD rephrased.The rephrased OOD exponent is again the flattest, consistent with width and depth scaling.
  • Cross-axis comparison: Across all axes, OOD rephrased is the flattest-scaling metric, while target-model scaling is steepest, followed by comparable depth and width scaling and then fact-count scaling.This indicates that robustness to linguistic variation is not easily addressed by scaling a single architectural or data dimension.
  • Comparison with fine-tuning: On validation, LoRA and full fine-tuning scale slightly better than hypernetworks, but hypernetworks have the steepest scaling on all three OOD splits.Hypernetwork exponents are −0.107 versus −0.083 for LoRA and −0.069 for full fine-tuning on OOD rephrased, and −0.171 versus −0.119 and −0.101 on OOD MCQ; the OOD gap widens with target size.

6 Conclusion and Future Work

The paper introduces MegaWikiQA and establishes systematic power-law scaling for hypernetwork-based knowledge injection across multiple architecture axes. It also identifies hypernetwork parameter overhead as a practical deployment limitation at larger scales.

  • Conclusion: MegaWikiQA supports the first systematic study of scaling laws for hypernetwork-based knowledge injection across width, depth, target model size, and injected-fact count.The study reports power-law improvements across all four axes.
  • Conclusion: Target model scaling produces the strongest gains, while hypernetworks generalize more effectively than matched-capacity finetuning methods in OOD settings.The reported trends extend across unseen domains and rephrased queries.
  • Future Work: 2.5B parameters made the highest-capacity hypernetwork nearly as large as the 1.5B-parameter target model, limiting practical deployability.This parameter overhead motivates future work on more parameter-efficient hypernetworks.

A Calculating Training Compute for Hypernetworks

Training compute is calculated by summing hypernetwork and target-model forward and backward costs, using parameter and token counts for each network. Depth scaling is effectively iso-compute because increasing hypernetwork depth changes per-example FLOPs by less than 2%.

  • Compute components: Training FLOPs comprise hypernetwork forward cost 2nhNh, target-model forward cost 2ntNt, target-model backward cost 2ntNt, and hypernetwork backward cost 4nhNh.Nh and Nt denote non-embedding hypernetwork and target-model parameters; nh and nt denote their input-token counts.
  • Token-count assumptions: The experiments fix target-model query length at nt = 15 and set hypernetwork input length to nh = 11.4 × Nfacts.For width, depth, and target-model scaling, Nfacts = 4 gives nh = 45.6.
  • Depth scaling compute: Increasing LHN from 1 to 16 increases Nh by only 1.9%, making depth scaling effectively iso-compute.Per-example FLOPs vary by less than 2% across the depth range because LoRA projection heads dominate Nh at dmodel = 256.
  • Scaling-axis accounting: The reported scaling experiments compare per-example FLOPs across hypernetwork width, depth, and injected fact-count axes.Tables 2–5 report these per-example FLOPs; the 52-fact run uses nh = 512 because of the hypernetwork context-window limit.

B Domain Classification

MegaWikiQA assigns each Wikidata5M-derived example to one of 39 semantic domains through a two-stage classification pipeline. The resulting labels support balanced multi-hop evaluation, with over 95% manual validation accuracy and known base-model advantages in held-out OOD domains.

  • Domain taxonomy: MegaWikiQA assigns each Wikidata5M triplet to one of 39 semantic domains for controlled cross-domain generalization evaluation.The taxonomy includes domains such as geography, political science, and music.
  • Domain taxonomy: The fixed 39-domain taxonomy draws partly on prior knowledge-editing benchmarks, and the classifier is trained on UniEdit- and GPT-4.1-derived triplet-domain pairs.This design promotes consistency with existing datasets and facilitates comparability.
  • Classification pipeline: Classification uses GPT-4.1 prediction followed by a fine-tuned LLM classifier head, trained from around 100K GPT-labelled samples collected with voting ensembles.The second stage validates the first-stage domain prediction.
  • Quality control: Over 95% manually verified classification accuracy and 92% agreement with GPT-4.1 predictions result from confidence filtering and quality control.Filtering low-confidence predictions reduces label noise for downstream evaluation.
  • Application to multi-hop examples: Labels propagate from single-hop triplets to multi-hop examples, enabling splits balanced by domain coverage and reasoning complexity.Reasoning complexity is measured by the number of hops, preventing distributional imbalance from confounding scaling experiments.
  • Domain-level base model accuracy: The held-out OOD domains—philosophy, linguistics, and civil engineering—have base-model accuracy approximately 4.8% above the in-distribution mean.This stronger prior knowledge affects interpretation of OOD results for the frozen target model.

C Dataset Statistics

MegaWikiQA uses disjoint evaluation splits with no entity or triplet overlap with training, while philosophy, linguistics, and civil engineering are fully withheld as OOD domains.

  • OOD domains: Philosophy, linguistics, and civil engineering are fully held out during training as OOD domains.These three domains are reserved for out-of-distribution evaluation.
  • Dataset overview: Table 6 summarizes the dataset statistics and all splits used in the experiments.The table provides the paper’s consolidated split-level dataset summary.
  • Split construction: All evaluation splits are disjoint from training at the triplet level, with zero entity and triplet overlap.This separation applies across the dataset splits used in the experiments.

D Hypernetwork Architecture Details … D.7 Target Model Scaling

The appendix specifies the hypernetwork’s encoding, transformer, LoRA generation, target-layer, training, tuning, and target-model-scaling procedures. Across target-model experiments, the hypernetwork is fixed while capacity varies from 0.5B to 14B parameters.

  • D Hypernetwork Architecture Details: The architecture details cover input encoding, transformer design, LoRA weight generation, target-layer selection, and hyperparameter tuning for scaling experiments.
  • D.1 Input Encoding: Facts are tokenized with the target model’s tokenizer, capped at 128 tokens, concatenated, and reduced to one representation through masked mean pooling.The pooling is parameter-free and treats non-padding token positions symmetrically.
  • D.2 Encoder Architecture: The transformer encoder uses RoPE on attention queries and keys, while post-norm residual blocks improve training stability across explored depths.
  • D.3 LoRA Weight Generation: Dedicated projection heads map the pooled representation to low-rank factors for each selected target weight matrix.Adaptations use r = 4, α = 8, and fixed magnitude scalar c = 0.01.
  • D.4 Target Layer Selection: LoRA adaptations target the later ⌊L/2⌋ layers, concentrating injection in higher-level semantic representations while keeping the adapted-layer fraction fixed across model sizes.
  • D.5 Training Objective: The hypernetwork minimizes answer-token cross-entropy end-to-end, receiving gradients through LoRA matrices while the target model remains frozen.
  • D.6 Hyperparameter Tuning: Learning rates are tuned with three anchor configurations—lowest, highest, and midpoint values—then interpolated using a fitted power law instead of exhaustive full-scale searches.A preliminary sweep found cosine annealing with AdamW consistently outperformed polynomial decay and Muon across tested learning rates.
  • D.7 Target Model Scaling: Target-model scaling compares Qwen2.5 models of 0.5B, 1.5B, 3B, 7B, and 14B parameters with a fixed hypernetwork architecture.Later ⌊L/2⌋ layers are adapted in each model, isolating target-model capacity from hypernetwork capacity.

E LoRA Rank Scaling

LoRA rank scaling was evaluated across ranks 2–64 with fixed hyperparameters and the same target model as the hypernetwork experiments. Unlike other scaling axes, performance saturates and is best modeled by a power law with an additive asymptotic loss floor.

  • Experimental setup: LoRA baselines were trained at ranks r ∈ {2, 4, 8, 16, 32, 64} with scaling factor α = 2r, holding other hyperparameters fixed.The experiments used the same target model as the hypernetwork experiments and plotted final-epoch loss against rank.
  • Scaling behavior: LoRA rank scaling exhibits a clear saturating regime and cannot be captured by a pure power law.The results are instead fit with L(r) = a·r^b+c, where c represents the asymptotic loss floor.
  • Scaling behavior: The fitted loss floors were 0.537 for validation, 0.562 for OOD non-rephrased, 0.800 for OOD rephrased, and 0.725 for OOD MCQ.These substantial floors imply diminishing returns from increasing adapter capacity beyond a modest rank.

F Loss Trajectories During Training

Loss trajectories are plotted against cumulative training FLOPs to compare configurations with different per-example compute costs. The experiments cover hypernetwork width, depth, fact count, and target-model scaling, tracking validation and OOD non-rephrased loss.

  • Trajectory setup: Validation and OOD non-rephrased loss are plotted against cumulative training FLOPs, enabling comparisons across configurations with different per-example compute costs.Darker colors denote larger configurations, including wider, deeper, larger-target, or more-fact settings.
  • Hypernetwork width scaling: Width trajectories compare dmodel ∈{64, 128, 256, 512, 1024} using validation and OOD non-rephrased loss.The figure reports both losses against cumulative training FLOPs.
  • Hypernetwork depth scaling: Depth trajectories compare LHN ∈{1, 2, 4, 8, 16} using validation and OOD non-rephrased loss.The narrow FLOPs range reflects the near iso-compute nature of depth scaling.
  • Fact count scaling: Fact-count trajectories compare Nfacts ∈{2, 4, 8, 16, 32, 52} using validation and OOD non-rephrased loss.More-fact configurations reach higher total FLOPs because of larger nh.
  • Target model scaling: Target-model trajectories compare sizes 0.5B, 1.5B, 3B, 7B, and 14B using validation and OOD non-rephrased loss.Larger target models incur higher per-example FLOPs and reach higher total FLOPs at the same number of epochs.
Loading 2607.19604v1…