An Emulator for Fine-Tuning Large Language Models using Small Language Models

• EFT decouples pre-training and fine-tuning scales, enabling direct attribution of capability changes and test-time modification of objectives without additional training.Source: §1 · p. 1, Figure 1 · p. 2
• EFT factorizes fine-tuned logits into base-model log probabilities and a behavior delta, then combines terms computed at different model sizes.Source: §1 · pp. 1–2
• Llama-2 EFT emulates 70B pre-training with 7B fine-tuning through Llama-2-base 70B + (Llama-2-chat 7B - Llama-2-base 7B).Source: §1 · p. 2
• Across model families and datasets, scaling pre-training primarily improves factual correctness, whereas scaling fine-tuning primarily improves helpfulness.Source: §1 · p. 2
• Up-scaling ensembles a small fine-tuned model with a large base model to approximate large-model fine-tuning without fine-tuning the large model.Source: §1 · p. 2

• Pre-training progressed from unsupervised representation learning to large transformer-based language models whose factual knowledge can be redirected through fine-tuning.These models support downstream linguistic tasks and general-purpose dialogue.Source: §2 · p. 2
• Increasing model scale improves pre-training benefits such as fluency, world knowledge, and reasoning ability.Source: §2 · pp. 2–3
• Up-scaling applies behavioral changes learned by a small fine-tuned model to the knowledge of a large pre-trained model, producing responses that combine user-intent following with factual correctness.The figure illustrates this by reweighting a large model’s factually correct conditional using a small-scale behavioral-change ratio.Source: Figure 2 · p. 3
• Contrastive decoding subtracts a scaled small-model log-probability from a large model, whereas this work interprets the difference as a log-importance weight without an added scaling hyperparameter.Source: §2 · p. 3
• EFT differs from related reward-reweighting methods by requiring no new reward model, grounding the procedure in reinforcement learning, and scaling efficiently with vocabulary size.Source: §2 · p. 3

3.1 Preliminaries

• EFT interprets fine-tuning as KL-constrained reinforcement learning, where the pre-trained model is the reference and β controls the constraint strength.The same formulation can map any fine-tuned model and its pre-trained model to a reward function.Source: §3.1 · pp. 3–4
• EFT factorizes a fine-tuned model into base log probabilities and a behavior delta, representing the change learned during fine-tuning.This partitioning supports independent scaling of pre-training knowledge and fine-tuning capabilities.Source: §3.1 · p. 4

3.2 Scale Decoupling with EFT

• Scale decoupling mixes knowledge from an N-scale pre-trained model with fine-tuning behavior from an M-scale model, with N > M yielding up-scaling and N < M yielding down-scaling.When N = M, the construction corresponds to sampling from the original policy.Source: §3.2 · p. 4
• EFT samples are generated from per-token conditionals using a timestep-wise approximation to the otherwise intractable sequence-level partition function.This approximation is related to temporally greedy interpretations of preference learning.Source: §3.2 · p. 4

3.3 Computational Factors and Language Model Up-Scaling

• Scaling pre-training primarily improves factuality, while scaling fine-tuning primarily improves perceived helpfulness; up-scaling achieves nearly 70% of the factuality gains relative to the small fine-tuned model alone.The aggregate result averages normalized improvements across Llama-1, Llama-2, and Falcon families and Anthropic-HH and ELI5 datasets.Source: Figure 3 · p. 5
• Up-scaling combines a large pre-trained model with a small fine-tuned model to approximate fine-tuning the large model, making EFT practically useful for improving small models.Unlike down-scaling, up-scaling does not require access to the large fine-tuned model and can approach its sampling cost when the fine-tuned model is much smaller.Source: §3.3 · p. 5

4.1 What Capabilities Arise from Scaling Pre-training vs Fine-tuning?

• Scaling pre-training primarily improves factuality, whereas scaling fine-tuning primarily improves perceived helpfulness; EFT up-scaling also consistently gains factuality and improves helpfulness.Results are normalized between the small and large fine-tuned models, whose scores provide approximate lower and upper bounds.Source: Figure 4 · p. 6, §4.1 · p. 7, Figure 6 · p. 8

4.2 EFT Enables Dynamic Test-Time Reward Interpolation

• EFT interpolates helpfulness and harmlessness rewards at sampling time, producing smooth tradeoff frontiers, while up-scaling yields a Pareto improvement without retraining.The method combines reward interpolation with a larger pre-trained model to emulate different fine-tuning tradeoffs.Source: §4.2 · p. 7, Figure 5 · p. 7

4.3 Efficient Sampling from Up-scaled Models with Speculative Decoding

• Speculative EFT decoding accelerates sampling from Llama-2-7B-chat up-scaled with Llama-2-70B-base by nearly 2.5x while producing identical samples.The small fine-tuned model proposes token blocks, which the large and small base models verify in parallel.Source: §4.3 · p. 8, Table 2 · p. 9

4.4 Conservative Decoding Strategies for Up-Scaled Models

• Top-p filtering of up-scaling weights produces mild improvements in factuality and helpfulness by preventing extreme reweighting of very unlikely tokens.The filter sets up-scaling weights to 1 within the small fine-tuned model’s top-p set rather than zeroing other tokens.Source: §4.4 · pp. 8–9, Table 1 · p. 8

4.3 Efficient Sampling from Up-scaled Models with Speculative Decoding

• Most tokens have small total variation distance between the small fine-tuned and up-scaled policies, supporting speculative decoding because only a few tokens receive substantial probability shifts.Figure 7 identifies the tokens with the largest up-weighting or down-weighting.Source: §4.3 · p. 8, Figure 7 · p. 9

4.5 Comparing GPT-4 Factuality Judgments with Human Evaluators

• Human and GPT-4 factuality labels agree 61% of the time, while author-checked disagreements find GPT-4 correct 77% of the time.The evaluation uses 100 ELI5 responses and examines 30 human–GPT-4 disagreements with author-collected ground-truth labels.Source: §4.5 · p. 9

5 Conclusion

• Most factuality gains from fine-tuning large pre-trained models can be acquired by combining a large base model with a small fine-tuned model through up-scaling.This approach emulates large-scale fine-tuning when that computation is prohibitive.Source: §5 · p. 9
• Emulated fine-tuning enables dynamic adjustment of behaviors such as helpfulness and harmlessness without additional training.Source: §5 · p. 9
• The methodology enables direct empirical exploration of scaling pre-training and fine-tuning independently.Source: §5 · p. 9

A.1 GPT-4 Prompts

• The appendix introduces GPT-4 prompts for evaluating factuality, helpfulness, and harmlessness of chatbot responses.Each prompt evaluates a response in the context of a user query.Source: §A.1 · p. 13, §A.1.2 · p. 14, §A.1.3 · p. 14

A.1.1 Factuality GPT-4 Prompt

• The factuality prompt asks evaluators to count inaccuracies critically and abstain from scoring unethical or illegal responses.Source: §A.1.1 · p. 13
• The factuality evaluation requires a reason followed by a single numerical score.Source: §A.1.1 · p. 13
• Figure 8 shows the Potato interface used for collecting human factuality labels.Source: Figure 8 · p. 14

A.1.2 Helpful GPT-4 Prompt

• The helpfulness prompt estimates the probability that an extremely critical user is completely satisfied that the response accomplishes their goals.It emphasizes that longer responses are not necessarily better.Source: §A.1.2 · p. 14
• The helpfulness output combines reasoning about the query and response with a 0–100 satisfaction probability.Source: §A.1.2 · p. 14

A.1.3 Harmless GPT-4 Prompt

• The harmlessness prompt scores responses from 0 to 100 according to their potential physical or emotional harm to the user or society.Source: §A.1.3 · p. 14
• The harmlessness format requires reasoning followed by a single 0–100 safety score.Source: §A.1.3 · p. 14

A.2 Human Annotation Collection

• Human factuality labels were collected on Prolific.co with the Potato annotation package, and labelers were compensated $15–18 per hour.The labeling interface is shown in Figure 8.Source: §A.2 · p. 14