---
title: "Scaling Laws for Hypernetwork-Based Knowledge Injection"
description: "New scaling laws for train-time knowledge injection: hypernetworks beat LoRA and full fine-tuning on out-of-distribution generalization, and the gap grows with target model size."
canonical: "https://nace.ai/research/scaling-law-for-hypernetwork"
updated: 2026-07-22
---

# Scaling Laws for Hypernetwork-Based Knowledge Injection

> New scaling laws for train-time knowledge injection: hypernetworks beat LoRA and full fine-tuning on out-of-distribution generalization, and the gap grows with target model size.

Research · Jul 22, 2026 · Nace AI

# Scaling Laws for Hypernetwork-Based Knowledge Injection

If you want an LLM to know something it doesn't already know (a company's internal docs, a specialized medical corpus, a set of recent regulations), you have a few options. You can stuff facts into the context window (in-context learning). You can fine-tune the base model, either fully or through LoRA. Or you can retrieve at inference time.

Each of these has known trade-offs. Context windows fill up. Fine-tuning is expensive and known to hurt out-of-distribution generalization. Retrieval punts the problem to a retriever.

We wanted to study a fourth option that gets less attention: **train a hypernetwork that reads a fact corpus and generates a LoRA adapter for a frozen target model.** The base model stays untouched. All gradient flow goes through the hypernetwork. Once trained, the hypernetwork produces an adapter that carries the corpus's knowledge in weight-space.

The idea of hypernetworks has been in academia for almost 30 years. But no one has studied how it scales and becomes practical. That's the question our paper answers.

## What we actually did

We built a benchmark called **MegaWikiQA**: 1.25M multi-hop question-answer pairs across 39 domains, deterministically generated from Wikidata5M. It's designed so that we can hold out entire domains for OOD evaluation, and so that the questions test compositional reasoning across multiple hops of the knowledge graph rather than single-fact recall.

Then we trained hypernetworks across four scaling axes:

- **Hypernetwork width** (`d_model` from 64 to 1024)
- **Hypernetwork depth** (1 to 16 transformer layers)
- **Target model size** (Qwen2.5 from 0.5B to 14B)
- **Number of injected facts per example** (2 to 64)

For every axis, we fit power laws to validation loss and three out-of-distribution splits: held-out domains, rephrased questions, and multiple-choice format.

The main finding is that **hypernetwork-based knowledge injection follows clean power-law scaling on every axis we tried**. Not surprising in hindsight (it's what most things in deep learning do), but this hadn't been established before, and the specific exponents are informative.

## Where the compute actually helps

Here's the summary of scaling exponents. More negative is steeper (better).

| Axis | ID Val | OOD Non-rep | OOD Rephrased | OOD MCQ |
| --- | --- | --- | --- | --- |
| HN width | -0.096 | -0.100 | -0.036 | -0.075 |
| HN depth | -0.088 | -0.096 | -0.042 | -0.063 |
| **Target model size** | **-0.226** | **-0.184** | **-0.107** | **-0.171** |
| Fact count | -0.080 | -0.077 | -0.028 | -0.077 |

Two things pop out:

**Scaling the target model matters far more than scaling the hypernetwork.** The target size exponent is roughly 2.5× steeper than the hypernetwork architecture exponents across the board. If you have a fixed budget and you're deciding whether to buy a bigger hypernetwork or a bigger frozen target model, buy the bigger target model.

**Depth and width scale nearly identically.** Their exponents are essentially the same within noise. We also verified that depth scaling is _iso-compute_, because the LoRA projection heads dominate parameter count at `d_model = 256`, going from 1 to 16 transformer layers only increases FLOPs by 1.9%. So the depth improvements aren't a compute confound; they're genuinely from architectural capacity.

## Comparison with Finetuning Methods

The scaling law story gets more interesting when we compare against fine-tuning at the same target model sizes.

Since target model scaling is by far the steepest axis for the hypernetwork, it's also the axis where the comparison against fine-tuning matters most. We ran two additional scaling laws over the same Qwen2.5 sizes: **LoRA fine-tuning** (r=16, α=32) and **full fine-tuning**.

Below are the three target-model scaling curves, one method per figure. Each plot shows in-distribution validation loss on the left and the three OOD metrics on the right (held-out domains / non-rephrased, rephrased, and multiple-choice), all on log–log axes vs target model size:

1. **Hypernetwork**: our method, scaling the frozen target from 0.5B to 14B
2. **LoRA fine-tuning**: same target sizes, adapters with r=16, α=32
3. **Full fine-tuning**: same setup; the 14B run was infeasible under our compute budget, so this fit covers 0.5B–7B

**Hypernetwork: loss vs. target size, log–log**

*(a) Validation loss, Validation loss by Target model parameters T*

| Target model parameters T | Validation (ID) |
| --- | ---: |
| 0.5B | 0.8211 |
| 1.5B | 0.5921 |
| 3B | 0.4958 |
| 7B | 0.4404 |
| 14B | 0.3815 |

- Fit, Validation (ID): y = 72 · x^-0.226

*(b) OOD generalization, OOD loss by Target model parameters T*

| Target model parameters T | OOD (non-rephrased) | OOD (rephrased) | OOD (MCQ) |
| --- | ---: | ---: | ---: |
| 0.5B | 0.7981 | 1.0383 | 0.9712 |
| 1.5B | 0.5691 | 0.7882 | 0.7311 |
| 3B | 0.4856 | 0.7403 | 0.6725 |
| 7B | 0.4677 | 0.7466 | 0.5909 |
| 14B | 0.4213 | 0.7012 | 0.5413 |

- Fit, OOD (non-rephrased): y = 29.7 · x^-0.184
- Fit, OOD (rephrased): y = 8.19 · x^-0.107
- Fit, OOD (MCQ): y = 28.6 · x^-0.171

*Hypernetwork target-model scaling, 0.5B–14B frozen targets: end-of-training validation loss and three OOD metrics with log-space power-law fits.*

**LoRA fine-tuning: loss vs. target size, log–log**

*(a) Validation loss, Validation loss by Target model parameters T*

| Target model parameters T | Validation (ID) |
| --- | ---: |
| 0.5B | 0.748034 |
| 1.5B | 0.543 |
| 3B | 0.449091 |
| 7B | 0.390528 |
| 14B | 0.316735 |

- Fit, Validation (ID): y = 110.16 · x^-0.250

*(b) OOD generalization, OOD loss by Target model parameters T*

| Target model parameters T | OOD (non-rephrased) | OOD (rephrased) | OOD (MCQ) |
| --- | ---: | ---: | ---: |
| 0.5B | 0.704767 | 0.890981 | 0.85847 |
| 1.5B | 0.571 | 0.801 | 0.732 |
| 3B | 0.495708 | 0.732541 | 0.656796 |
| 7B | 0.472048 | 0.71193 | 0.628907 |
| 14B | 0.41856 | 0.674786 | 0.570287 |

- Fit, OOD (non-rephrased): y = 14.15 · x^-0.151
- Fit, OOD (rephrased): y = 4.59 · x^-0.083
- Fit, OOD (MCQ): y = 9.10 · x^-0.119

*LoRA fine-tuning (r=16, α=32) target-model scaling, 0.5B–14B: same axes and splits as the hypernetwork figure.*

**Full fine-tuning: loss vs. target size, log–log**

*(a) Validation loss, Validation loss by Target model parameters T*

| Target model parameters T | Validation (ID) |
| --- | ---: |
| 0.5B | 0.623065 |
| 1.5B | 0.469 |
| 3B | 0.391792 |
| 7B | 0.324136 |

- Fit, Validation (ID): y = 90.62 · x^-0.249

*(b) OOD generalization, OOD loss by Target model parameters T*

| Target model parameters T | OOD (non-rephrased) | OOD (rephrased) | OOD (MCQ) |
| --- | ---: | ---: | ---: |
| 0.5B | 0.691165 | 0.880826 | 0.833306 |
| 1.5B | 0.56 | 0.808 | 0.736 |
| 3B | 0.490049 | 0.756956 | 0.675314 |
| 7B | 0.42783 | 0.738325 | 0.64216 |

- Fit, OOD (non-rephrased): y = 26.82 · x^-0.183
- Fit, OOD (rephrased): y = 3.51 · x^-0.069
- Fit, OOD (MCQ): y = 6.27 · x^-0.101

*Full fine-tuning target-model scaling: fits cover 0.5B–7B; the 14B run was infeasible under our compute budget.*

Read across the three plots and the pattern is consistent. For each curve we fit a power law of the form

$$
L(T) = a \cdot T^{\alpha}
$$

where _T_ is target-model parameter count, _L_ is loss on a given split, _a_ is a fitted scale constant, and **α** is the scaling exponent. More negative α means loss falls faster as the target model grows; steeper is better. The table below reports those fitted α values for each method and evaluation split.

On average, hypernetworks turned out about **3–4% better on OOD settings** than the fine-tuning baselines under matched conditions. The exponents:

| Method       | ID Val     | OOD Non-rep | OOD Rephrased | OOD MCQ    |
| ------------ | ---------- | ----------- | ------------- | ---------- |
| Hypernetwork | -0.226     | **-0.184**  | **-0.107**    | **-0.171** |
| LoRA FT      | **-0.250** | -0.151      | -0.083        | -0.119     |
| Full FT      | -0.249     | -0.183      | -0.069        | -0.101     |

Fine-tuning methods scale in-distribution validation _slightly_ better than the hypernetwork. That makes sense since direct parameter updates fit the training distribution more aggressively.

But **the hypernetwork wins every OOD axis**, and the win widens with target model size. On OOD rephrased (the metric that tests whether the model has actually generalized versus overfit to grammar-templated queries), the hypernetwork's exponent is 1.3–1.5× steeper than LoRA and full FT. On OOD multiple-choice, similar story: -0.171 versus -0.119 (LoRA) and -0.101 (full FT).

The gap isn't a fixed offset. It grows with scale. If you extrapolate the trends, hypernetwork adaptation looks increasingly favorable exactly in the regime where deployment matters most.

## LoRA rank scaling saturates. Target scaling doesn't.

One more piece worth flagging: we also scaled LoRA rank from r=2 to r=64. Unlike every other scaling axis in the paper, **LoRA rank scaling saturates**. The best fit isn't a pure power law; it's a power law with an additive constant (an asymptotic floor).

The floor is high. Even at r=64, LoRA can't close the OOD gap to the hypernetwork through capacity alone. Rank isn't the axis that helps. Target model size is.

## Honest limitations

**Hypernetwork parameter count.** At the largest configurations, the hypernetwork approaches the target model in parameter count. Getting most of the OOD benefit at a fraction of the hypernetwork size is the natural next efficiency question.

## Why this matters

If you're building anything that needs LLMs to actually know a large body of specific facts (enterprise search over internal docs, medical Q&A over patient records, legal reasoning over jurisdiction-specific case law), the question of _how_ you get that knowledge into the model is a live one. Fine-tuning is the default answer. Our results suggest that a train-time hypernetwork is a scalable alternative with better OOD behavior, and that the advantage widens as base models get bigger.

**Paper:** [arXiv](https://arxiv.org/abs/2607.19604) **Dataset:** [MegaWikiQA on Hugging Face](https://huggingface.co/collections/nace-ai/hypernetwork-datasets)

## For agents

- [Drex API guide](https://nace.ai/drex.md): the decision model: request shape, question types, benchmarks and deployment, as markdown
- [Decision endpoint](https://console.nace.ai/v1/systemone): POST with model "drex-latest", a state string and typed questions (choice, noul, score); https://console.nace.ai/v1/models lists checkpoints
- [Document endpoints](https://console.nace.ai/v1/documents/{parse|split|classify|extract|ground}): POST a source (a public URL, an uploaded workspace file or a finished parse job) and poll GET https://console.nace.ai/v1/documents/jobs/{id}; the same key and credit as decisions
- [Console](https://console.nace.ai): API keys, the playground, usage and billing; send `Authorization: Bearer $NACE_API_KEY` on every request
- [Console docs](https://console.nace.ai/docs): quickstart, question types, the document routes, errors, limits and pricing
- [Nace Document Intelligence](https://nace.ai/ndi.md): the Perception series: parse, split, classify, extract and ground any file type with a location on every value
- [Developers](https://nace.ai/developers.md): The execution layer for agents: Perception, Decision and Context behind one key, the Drex Max frontier, install steps, a scripted agent session and the four deployment boundaries
- [Site index](https://nace.ai/llms.txt): every page on nace.ai; append `.md` to any page URL or send `Accept: text/markdown` for the markdown version