NEFTune: Noisy Embeddings Improve Instruction Finetuning

1 INTRODUCTION

• Instruction fine-tuning is crucial for making LLMs useful, and performance depends substantially on extracting value from small curated instruction datasets.Earlier FLAN and T0 work focused on cross-task generalization, while real-world use also requires free-form answers to open-ended queries.Source: §1 · p. 1, §1.1 · p. 2
• 29.8% to 64.7%: noisy embeddings raise LLaMA-2-7B’s AlpacaEval score, while the paper reports maintained factual question-answering performance.The authors describe the augmentation as adding no compute or data overhead.Source: §1 · p. 1, Figure 1 · p. 1

1.1 RELATED WORK

• Instruction-following datasets include Self-Instruct-derived Alpaca, ChatGPT-evolved instructions, and ShareGPT conversations collected from users.Other work described here curates STEM and reasoning data or filters examples for quality.Source: §1.1 · p. 2
• Prior noisy-input methods include adversarial embedding perturbations, whereas NEFTune uses non-adversarial noise and adopts earlier noise-scaling rules.The passage also notes noisy-input applications in graph models, image captioning, and differential privacy.Source: §1.1 · p. 2

2 NEFTU N E: NOISY EMBEDDING INSTRUCTION FINETUNING

• NEFTune adds a random noise vector to instruction-token embeddings before the model makes its prediction during fine-tuning.The noise entries are independently sampled from a uniform range of [−1, 1].Source: §2 · p. 2
• The approach is non-adversarial: it samples random noise rather than optimizing perturbations to maximize their effect on model performance.The paper describes Algorithm 1 as the detailed procedure.Source: §2 · p. 2
• Noise is scaled using sequence length L, embedding dimension d, and tunable parameter α.The scaling rule is borrowed from adversarial machine-learning work.Source: §2 · p. 2

3.1 MODELS

• Most experiments used 7B-parameter LLaMA-1, LLaMA-2, and OPT-6.7B models; separate LLaMA-2 runs used 13B and 70B parameters, and RLHF models were improved by fine-tuning LLaMA-2-Chat (7B).Source: §3.1 · pp. 2–3

3.2 INSTRUCTION FINETUNING DATASETS

• The datasets are Alpaca, 70K single-turn Evol-Instruct examples, 25K curated Open-Platypus questions, and 70K ShareGPT conversations adapted toward single-turn format.The study uses only single-turn data because of hardware memory constraints.Source: §3.2 · p. 3
• Default training uses an Alpaca system prompt except for ShareGPT, which uses the Vicuna prompt; hyperparameters are selected through a coarse LLaMA-1 Alpaca sweep.The sweep is reported to improve over the standard Alpaca model by 6%.Source: §3.2 · p. 3

3.3 EVALUATION

• AlpacaEval measures conversational win rate against Text-Davinci-003, while ARC, HellaSwag, MMLU, and TruthfulQA assess factual and reasoning capabilities.The evaluation section uses GPT-4 and ChatGPT as judges; Figure 2 compares win rates across models and datasets, and Table 1 reports a 15% average improvement.Source: §3.3 · pp. 3–4, Figure 2 · p. 4, Table 1 · p. 4

4 RESULTS

• 15.1% average increase in AlpacaEval performance across all studied 7B datasets accompanies NEFTune training.The passage also reports improvements on older LLaMA-1 and OPT models.Source: §4 · p. 4
• 88.81% Win Rate versus 75.03% (+13.78%) follows NEFTune on the Evol-Instruct-trained LLaMA-2 70B model.Table 2 covers further finetuning of LLaMA-2-Chat (7B) and LLaMA-2 (13B and 70B).Source: §4 · p. 4, Table 2 · p. 5
• 10% additional performance increase is reported for LLaMA-2-Chat after NEFTune, though some capabilities such as refraining from toxic output may be affected.The model had already undergone multiple rounds of RLHF.Source: §4 · p. 4
• OpenLLM Leaderboard scores remain stable across the evaluated datasets and models, indicating no observed loss on these tasks.The evaluation covers MMLU, ARC, HellaSwag, and TruthfulQA.Source: §4 · p. 5, Figure 3 · p. 5
• NEFTune improves AlpacaEval Win Rate across all studied model sizes and datasets with QLoRA, but gains are milder than with full finetuning.The authors suggest different epoch settings or 4-bit quantization may explain the smaller gains.Source: §4 · p. 5, Table 3 · p. 6
• In a quantum-computing example, the NEFTune response is more fluid and expands explanations of superposition and entanglement while mentioning applications.The example is from LLaMA-2 trained on Alpaca.Source: §4 · p. 6

5.1 OVERFITTING

• NEFT yields higher Alpaca training loss but slightly lower Evol-Instruct test loss, lower training-response ROUGE-L and BLEU, and less overfitting without reduced test performance.The authors interpret the lower ROUGE-L and BLEU as less reproduction of training-response wording.Source: Figure 4 · p. 7, §5.1 · p. 7, Figure 5 · p. 8

5.2 LENGTH VERSUS TOKEN DIVERSITY

• NEFT models generate longer outputs while keeping 2-gram repetition rates and overall token log-diversity nearly identical.Diversity was measured on fixed-length initial chunks, with dataset-specific cutoffs.Source: §5.2 · pp. 7–8, Table 4 · p. 8

5.3 LENGTH IS (NOT) ALL YOU NEED

• Prompting for longer responses boosts AlpacaEval scores by 16%, but forced 250-token completions improve only marginally and generation-time strategies do not match NEFTune.Gaussian noise produces longer outputs without improved performance; longer generations do not consistently predict AlpacaEval win rate.Source: §5.3 · p. 8, Table 5 · p. 9, Table 6 · p. 9

5.4 HUMAN STUDY

• Human annotators preferred NEFT in 88 of 140 comparisons, yielding a 74.6% win score; a modified GPT-4 comparison gave a 92.80% win score.Annotators compared responses from Alpaca-finetuned LLaMA-2 models with and without NEFT in random order.Source: §5.4 · pp. 8–9

6 CONCLUSIONS AND LIMITATIONS

• NEFTune’s gains and the tendency to overfit small instruction datasets lead the authors to argue that regularization deserves renewed attention in LLM training.Source: §6 · p. 9
• The study relies centrally on AlpacaEval, which is subject to the biases of a single GPT-4 judge, and does not validate 70B variants across multiple datasets.Limited compute also required fixed hyperparameters for most NEFTune runs rather than sweeps.Source: §6 · p. 9
• Despite its empirical studies, the paper does not conclusively explain why NEFTune works.Source: §6 · p. 9

7 ETHICS STATEMENT

• The study does not rigorously evaluate safety and reliability characteristics such as toxicity or refusal of potentially harmful responses.Source: §7 · p. 10

8 REPRODUCIBILITY STATEMENT

• Experiments used commodity-level CPUs and GPUs running open-source software, except for 70B-parameter finetuning.Source: §8 · p. 10
• The reproducibility statement names contributors and summarizes roles including code, experiments, method development, evaluation, and writing.Source: AUTHOR CONTRIBUTIONS · p. 10

A.1 HYPERPARAMETERS

• For 70B models, NEFTune used α = 15; average output character length rose from 852 to 1241 (+389).Other potentially better noise settings were not explored due to computational constraints.Source: §A.1 · p. 13
• The 7B models used three epochs, effective batch size 128, and learning rate 5e-5 with Adam.Training used bfloat16 precision; 7B models ran on four A5000s and 13B models on eight.Source: §A.1 · p. 13
• Noise runs tested L2 balls of 5, 10, and 15, reporting the best AlpacaEval result using ChatGPT as evaluator.The 7B models used sequence length 512; 70B models used sequence length 2048, batch size 64, weight decay 0.1, and learning rate 2e−5.Source: §A.1 · p. 13
• Further finetuning can improve LLaMA-2-Chat (7B) and LLaMA-2 (13B).Table 9 reports this comparison.Source: Table 9 · p. 13

A.2 ADDITIONAL ABLATION STUDIES

• Uniform noise performs slightly better than Gaussian noise.The study also compares NEFTune with different trainable-parameter configurations; the top two rows train all parameters.Source: §A.2 · p. 13, Table 11 · p. 14
• Decoding hyperparameter ablations show minimal performance variation.The study uses greedy decoding with a repetition penalty of 1.2.Source: §A.2 · p. 13, Table 10 · p. 14
• Performance plateaus at higher finetuning epoch counts.Table 14 reports AlpacaEval ChatGPT win rate as a function of epochs.Source: §A.2 · p. 13

A.3 ADDITIONAL ANALYSIS

• No sentence tokens flipped up to α = 15, while flips appeared at α ≥ 25.Figure 6 plots the percentage of tokens per sentence flipped across noise levels; all reported paper results use α ≤ 15.Source: §A.3 · p. 14, Figure 6 · p. 14
• NEFT performs better than FreeLB.This comparison is reported in Table 12.Source: Table 12 · p. 15
• On ShareGPT, longer sequences produce a slight performance increase over standard finetuning at the same sequence length.Table 13 reports this comparison using ChatGPT as evaluator.Source: Table 13 · p. 15
• For a given LLaMA-1 or LLaMA-2 base model, the top-100 singular-value distribution stayed unchanged across variants with and without NEFTune.Figure 7 shows the top 100 eigenvalues of embedding similarity matrices across models.Source: §A.3 · p. 15, Figure 7 · p. 15
• Even at high noise, noised embeddings flipped to a different nearest neighbor for at most 0.4% of vocabulary tokens.The analysis varies noise scale α and sequence-length scaling factor L; the authors conclude token semantic relationships are not changed.Source: §A.3 · p. 15, Figure 8 · p. 16
• The authors interpret NEFTune’s effects as mostly intra-token perturbations, although some LLaMA-2 Alpaca embeddings flipped even at the lowest noise level studied.Figure 9 shows flipped tokens at α = 1 and sequence length 512 in LLaMA-2.Source: §A.3 · p. 15, Figure 9 · p. 16

• For the quantum-computing question, Alpaca-NEFT gives a substantially longer explanation than Alpaca, while the ShareGPT and ShareGPT-NEFT responses are identical.The Alpaca-NEFT answer adds discussion of qubits, algorithms, applications, and entanglement; the ShareGPT pair both give a short explanation of quantum computers’ potential to solve some problems faster.Source: §A.4 · p. 17
• The Evolve-Instruct quantum-computing responses share an opening explanation, but the NEFT version continues with discussion of quantum gates, factorization, and error correction.The supplied NEFT response continues into a discussion of gates and practical challenges, while the baseline response discusses parallel computation, entanglement, and applications.Source: §A.4 · pp. 18–19
• For OpenPlatypus, the baseline begins with a brief description of quantum computing, whereas the NEFT response provides a longer account of superposition, entanglement, challenges, and applications.The baseline description spans the start of the quantum-computing response; the NEFT account continues across the next passage.Source: §A.4 · pp. 19–20
• In the symphony-review example, Alpaca-NEFT adds emotional reactions and conductor and audience details beyond Alpaca’s general praise of the orchestra and concert.The baseline describes the orchestra, acoustics, and lighting; the NEFT response describes the conductor’s gestures and audience reactions.Source: §A.4 · p. 20
• The Evolve-Instruct and OpenPlatypus symphony responses describe different concert programs and performance details in their baseline and NEFT versions.The Evolve-Instruct pair names different symphonies and composers; the OpenPlatypus pair likewise describes different works and soloists.Source: §A.4 · pp. 21–24

A.4 QUALITATIVE EXAMPLES

• NEFT’s Van Gogh response develops multiple speculative scenarios, adding possibilities about artistic exploration, relationships, mental illness, and his enduring legacy.The baseline answer focuses mainly on continued artistic output and a changed legacy.Source: §A.4 · pp. 24–25
• For the Broadway question, NEFT lists actors with career details and productions, while the baseline provides names alone.NEFT’s examples include Al Pacino, Meryl Streep, James Earl Jones, Nathan Lane, and Bette Midler.Source: §A.4 · p. 25
• Both Evolve-Instruct models give incorrect answers to the polynomial evaluation question.The paper identifies this as a case where neither model outputs the correct answer.Source: §A.4 · p. 25