Scaling Laws for Hypernetwork-Based Knowledge Injection

If you want an LLM to know something it doesn't already know — a company's internal docs, a specialized medical corpus, a set of recent regulations — you have a few options. You can stuff facts into the context window (in-context learning). You can fine-tune the base model, either fully or through LoRA. Or you can retrieve at inference time.

Each of these has known trade-offs. Context windows fill up. Fine-tuning is expensive and known to hurt out-of-distribution generalization. Retrieval punts the problem to a retriever.

We wanted to study a fourth option that gets less attention: train a hypernetwork that reads a fact corpus and generates a LoRA adapter for a frozen target model. The base model stays untouched. All gradient flow goes through the hypernetwork. Once trained, the hypernetwork produces an adapter that carries the corpus's knowledge in weight-space.

The idea of hypernetworks has been in academia for almost 30 years. But no one has studied how it scales and becomes practical. That's the question our paper answers.

What we actually did

We built a benchmark called MegaWikiQA: 1.25M multi-hop question-answer pairs across 39 domains, deterministically generated from Wikidata5M. It's designed so that we can hold out entire domains for OOD evaluation, and so that the questions test compositional reasoning across multiple hops of the knowledge graph rather than single-fact recall.

Then we trained hypernetworks across four scaling axes:

• Hypernetwork width (d_model from 64 to 1024)
• Hypernetwork depth (1 to 16 transformer layers)
• Target model size (Qwen2.5 from 0.5B to 14B)
• Number of injected facts per example (2 to 64)

For every axis, we fit power laws to validation loss and three out-of-distribution splits: held-out domains, rephrased questions, and multiple-choice format.

The main finding is that hypernetwork-based knowledge injection follows clean power-law scaling on every axis we tried. Not surprising in hindsight — it's what most things in deep learning do — but this hadn't been established before, and the specific exponents are informative.

Where the compute actually helps

Here's the summary of scaling exponents. More negative is steeper (better).

Two things pop out:

Scaling the target model matters far more than scaling the hypernetwork. The target size exponent is roughly 2.5× steeper than the hypernetwork architecture exponents across the board. If you have a fixed budget and you're deciding whether to buy a bigger hypernetwork or a bigger frozen target model, buy the bigger target model.

Depth and width scale nearly identically. Their exponents are essentially the same within noise. We also verified that depth scaling is iso-compute — because the LoRA projection heads dominate parameter count at d_model = 256, going from 1 to 16 transformer layers only increases FLOPs by 1.9%. So the depth improvements aren't a compute confound; they're genuinely from architectural capacity.

Comparison with Finetuning Methods

The scaling law story gets more interesting when we compare against fine-tuning at the same target model sizes.

Since target model scaling is by far the steepest axis for the hypernetwork, it's also the axis where the comparison against fine-tuning matters most. We ran two additional scaling laws over the same Qwen2.5 sizes: LoRA fine-tuning (r=16, α=32) and full fine-tuning.

Below are the three target-model scaling curves, one method per figure. Each plot shows in-distribution validation loss on the left and the three OOD metrics on the right (held-out domains / non-rephrased, rephrased, and multiple-choice), all on log–log axes vs target model size:

1. Hypernetwork — our method, scaling the frozen target from 0.5B to 14B
2. LoRA fine-tuning — same target sizes, adapters with r=16, α=32
3. Full fine-tuning — same setup; the 14B run was infeasible under our compute budget, so this fit covers 0.5B–7B

Hypernetwork: loss vs. target size, log–log

(a) Validation loss

(b) OOD generalization

Validation (ID):y = 72 · T^-0.226OOD (non-rephrased):y = 29.7 · T^-0.184OOD (rephrased):y = 8.19 · T^-0.107OOD (MCQ):y = 28.6 · T^-0.171

LoRA fine-tuning: loss vs. target size, log–log

(a) Validation loss

(b) OOD generalization

Validation (ID):y = 110.16 · T^-0.250OOD (non-rephrased):y = 14.15 · T^-0.151OOD (rephrased):y = 4.59 · T^-0.083OOD (MCQ):y = 9.10 · T^-0.119

Full fine-tuning: loss vs. target size, log–log

(a) Validation loss

(b) OOD generalization

Validation (ID):y = 90.62 · T^-0.249OOD (non-rephrased):y = 26.82 · T^-0.183OOD (rephrased):y = 3.51 · T^-0.069OOD (MCQ):y = 6.27 · T^-0.101

Read across the three plots and the pattern is consistent. For each curve we fit a power law of the form

$$L(T) = a \cdot T^{\alpha}$$

where T is target-model parameter count, L is loss on a given split, a is a fitted scale constant, and α is the scaling exponent. More negative α means loss falls faster as the target model grows — steeper is better. The table below reports those fitted α values for each method and evaluation split.

On average, hypernetworks turned out about 3–4% better on OOD settings than the fine-tuning baselines under matched conditions. The exponents:

Fine-tuning methods scale in-distribution validation slightly better than the hypernetwork. That makes sense since direct parameter updates fit the training distribution more aggressively.

But the hypernetwork wins every OOD axis, and the win widens with target model size. On OOD rephrased — the metric that tests whether the model has actually generalized versus overfit to grammar-templated queries — the hypernetwork's exponent is 1.3–1.5× steeper than LoRA and full FT. On OOD multiple-choice, similar story: -0.171 versus -0.119 (LoRA) and -0.101 (full FT).

The gap isn't a fixed offset. It grows with scale. If you extrapolate the trends, hypernetwork adaptation looks increasingly favorable exactly in the regime where deployment matters most.

LoRA rank scaling saturates. Target scaling doesn't.

One more piece worth flagging: we also scaled LoRA rank from r=2 to r=64. Unlike every other scaling axis in the paper, LoRA rank scaling saturates. The best fit isn't a pure power law — it's a power law with an additive constant (an asymptotic floor).

The floor is high. Even at r=64, LoRA can't close the OOD gap to the hypernetwork through capacity alone. Rank isn't the axis that helps. Target model size is.

Honest limitations

Hypernetwork parameter count. At the largest configurations, the hypernetwork approaches the target model in parameter count. Getting most of the OOD benefit at a fraction of the hypernetwork size is the natural next efficiency question.

Why this matters

If you're building anything that needs LLMs to actually know a large body of specific facts — enterprise search over internal docs, medical Q&A over patient records, legal reasoning over jurisdiction-specific case law — the question of how you get that knowledge into the model is a live one. Fine-tuning is the default answer. Our results suggest that a train-time hypernetwork is a scalable alternative with better OOD behavior, and that the advantage widens as base models get bigger.

Paper: arXiv Dataset: MegaWikiQA on Hugging Face