PaperlayerView exact revision on arXiv

Source-linked AI summary

Nemotron-4 340B Technical Report

Nvidia, :, Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong, Daniel Egert, Ellie Evans, Aleksander Ficek, Denys Fridman, Shaona Ghosh, Boris Ginsburg, Igor Gitman, Tomasz Grzegorzek, Robert Hero, Jining Huang, Vibhu Jawa, Joseph Jennings, Aastha Jhunjhunwala, John Kamalu, Sadaf Khan, Oleksii Kuchaiev, Patrick LeGresley, Hui Li, Jiwei Liu, Zihan Liu, Eileen Long, Ameya Sunil Mahabaleshwarkar, Somshubra Majumdar, James Maki, Miguel Martinez, Maer Rodrigues de Melo, Ivan Moshkov, Deepak Narayanan, Sean Narenthiran, Jesus Navarro, Phong Nguyen, Osvald Nitski, Vahid Noroozi, Guruprasad Nutheti, Christopher Parisien, Jupinder Parmar, Mostofa Patwary, Krzysztof Pawelec, Wei Ping, Shrimai Prabhumoye, Rajarshi Roy, Trisha Saar, Vasanth Rao Naik Sabavat, Sanjeev Satheesh, Jane Polak Scowcroft, Jason Sewall, Pavel Shamis, Gerald Shen, Mohammad Shoeybi, Dave Sizer, Misha Smelyanskiy, Felipe Soares, Makesh Narsimhan Sreedhar, Dan Su, Sandeep Subramanian, Shengyang Sun, Shubham Toshniwal, Hao Wang, Zhilin Wang, Jiaxuan You, Jiaqi Zeng, Jimmy Zhang, Jing Zhang, Vivienne Zhang, Yian Zhang, Chen Zhu

arXiv:2406.11704v2cs.CLcs.AIcs.LG

TL;DR

High-quality human data is time-consuming and costly to collect, motivating exploration of synthetic data generation for model alignment. The paper releases Base, Instruct, and Reward models and describes a synthetic-data pipeline for prompt, response, and dialogue generation, quality filtering, and preference ranking. Nemotron-4-340B-Instruct surpasses corresponding instruct models; Nemotron-4-340B-Reward achieves top RewardBench accuracy as of publication (Allen AI, 2024); over 98% of Instruct training data was synthetic throughout alignment.

  • Problem

    High-quality human data is time-consuming and costly to collect, motivating exploration of synthetic data generation for model alignment.

    Source: §3.2 · p. 6

  • Method

    The paper releases Base, Instruct, and Reward models and describes a synthetic-data pipeline for prompt, response, and dialogue generation, quality filtering, and preference ranking.

    Source: §1 · pp. 1–2

  • Results

    Nemotron-4-340B-Instruct surpasses corresponding instruct models; Nemotron-4-340B-Reward achieves top RewardBench accuracy as of publication (Allen AI, 2024); over 98% of Instruct training data was synthetic throughout alignment.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The models and pipeline are released to support community research and AI applications, including creating high-quality data adaptable to a wide range of domains.

    Source: §1 · p. 2

    Nemotron-4-340B was susceptible to existing jailbreaks, with a pass rate below 30% for attempted jailbreaks.

    Source: §3.4.3 · p. 19

Abstract

from arXiv · show

We release the Nemotron-4 340B model family, including Nemotron-4-340B-Base, Nemotron-4-340B-Instruct, and Nemotron-4-340B-Reward. Our models are open access under the NVIDIA Open Model License Agreement, a permissive model license that allows distribution, modification, and use of the models and its outputs. These models perform competitively to open access models on a wide range of evaluation benchmarks, and were sized to fit on a single DGX H100 with 8 GPUs when deployed in FP8 precision. We believe that the community can benefit from these models in various research studies and commercial applications, especially for generating synthetic data to train smaller language models. Notably, over 98% of data used in our model alignment process is synthetically generated, showcasing the effectiveness of these models in generating synthetic data. To further support open research and facilitate model development, we are also open-sourcing the synthetic data generation pipeline used in our model alignment process.

1 Introduction

The introduction presents the Nemotron-4 340B family as open-access models with competitive benchmark performance and a synthetic-data pipeline intended to support research and model development. It describes alignment using supervised and preference fine-tuning, with synthetic data providing most of the alignment data.

1 Introduction

  • The Nemotron-4 340B base model was trained on 9 trillion tokens from a high-quality dataset.The introduction contrasts this with prior efforts to increase model accuracy through training on more, higher-quality tokens.Source: §1 · p. 1
  • Alignment proceeds from supervised fine-tuning to preference fine-tuning, and the reward model supports response-quality assessment, filtering, and preference ranking.The introduction names RLHF and DPO as preference fine-tuning approaches.Source: §1 · p. 1
  • Base is competitive with open-access base models on commonsense benchmarks, Instruct surpasses corresponding instruct models, and Reward achieves top RewardBench accuracy as of publication.The comparison covers ARC-Challenge, MMLU, and BigBench Hard for Base; instruction following and chat for Instruct; and RewardBench for Reward.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Over 98% of Nemotron-4-340B-Instruct alignment data was synthetically generated, using a pipeline for prompt, response, and dialogue generation, quality filtering, and preference ranking.The pipeline is designed to support both supervised fine-tuning and preference fine-tuning.Source: §1 · p. 2
  • The authors aim to broaden access for research and AI applications while supporting responsible LLM use, and state they do not intend harmful-content generation.Source: §1 · p. 2
  • The three models are released under a permissive license, alongside training and inference code, generation prompts, a human-annotated preference dataset, and the Reward model.The license is described as permissive for commercial applications.Source: §1 · pp. 2–3

2 Pretraining

Nemotron-4-340B-Base combines a 9T-token pretraining run with a decoder-only Transformer architecture and distributed training. It achieves the strongest accuracy on commonsense reasoning tasks and BBH, and is competitive on MMLU and HumanEval.

2.1 Data

  • The 9T-token pretraining blend contains 70% English, 15% multilingual, and 15% source-code data, spanning 53 natural languages and 43 programming languages.The first 8T tokens were formal pretraining; the final 1T were continued training.Source: §2.1 · p. 3

2.2 Architectural Details

  • The model uses a decoder-only Transformer with causal attention, RoPE, SentencePiece tokenization, squared ReLU activations, GQA, and untied input-output embeddings.It has 9.4 billion embedding parameters and 331.6 billion non-embedding parameters.Source: §2.2 · p. 3

2.3 Training Details

  • Training combined 8-way tensor parallelism, 12-way interleaved pipeline parallelism, data parallelism, and a distributed optimizer that shards optimizer state.The run used 768 DGX H100 nodes; the batch-size ramp schedule and time and efficiency metrics are summarized in Table 2.Source: §2.3 · p. 4, Table 2 · p. 4
  • After 8T tokens, continued training used 1T additional tokens and shifted toward higher-quality sources, selected QA examples, and data from low-accuracy areas.The authors report that switching the data distribution and learning-rate decay schedule significantly improves model quality; the final schedule prioritizes a steeper decay slope.Source: §2.3 · p. 4

2.4 Base Model Evaluation

  • Nemotron-4-340B-Base achieves the strongest accuracy on commonsense reasoning tasks and BBH, while remaining competitive on MMLU and HumanEval.Source: §2.4 · p. 5, Table 3 · p. 5

3 Alignment

This section describes Nemotron-4’s alignment approach, combining synthetic-data-driven supervised and preference fine-tuning with reward modeling, and evaluates its performance, human comparisons, safety, and limitations.

3.1 Reward Modeling

  • The multi-attribute regression reward model achieved the highest RewardBench accuracy at publication and predicts five HelpSteer attributes for weighted aggregation into an overall reward.The attributes are Helpfulness, Correctness, Coherence, Complexity, and Verbosity; regression is presented as better at disentangling helpfulness from artifacts such as response length.Source: §3.1 · pp. 5–6

3.2 Alignment Data

  • Over 98% of supervised-fine-tuning and preference-fine-tuning data was synthesized, while about 20K human-annotated examples supported alignment.For preference triplets, ground-truth labels or verifiers judge answerable tasks, while Reward-Model-as-Judge scored 0.87 versus 0.54 average accuracy for LLM-as-Judge on Chat-Hard.Source: §3.2 · p. 6, §3.2.3 · pp. 9–10

3.2.1 Prompt Preparation

  • Prompt preparation spans diverse tasks, topics, and instruction formats; synthetic dialogues use three turns, role-playing, and reward-model quality filtering.LMSYS prompts were split without overlap between supervised and preference learning, with potentially unsafe prompts removed only from the supervised split; their average helpfulness was lower than for synthetic prompts.Source: §3.2.1 · pp. 6, 8, §3.2.2 · p. 9

3.2.4 Iterative Weak-to-Strong Alignment

  • Iterative weak-to-strong alignment let models trained on generated data surpass their initial teacher, while stronger generators produced higher-quality data for later rounds.The first iteration used Mixtral-8x7B-Instruct-v0.1; the resulting intermediate instruct model then served as the generator for the second iteration.Source: §3.2.4 · pp. 10–11

3.2.5 Additional Data Sources

  • Supplementary training data targets topic focus, refusal on incapable tasks, STEM and document-grounded reasoning, and function calling.Topic-following data includes distractor turns; incapable-task examples pair questions requiring unavailable capabilities with prompted rejection responses.Source: §3.2.5 · p. 11

3.3.1 Staged Supervised Fine-tuning

  • Two-stage SFT addresses conflicts among concurrently learned behaviors by training on coding data first and a general task blend second.Genetic Instruct expanded high-quality seeds into a curated dataset of approximately 800K coding samples for the first stage; the general blend retained 2% code samples to mitigate forgetting.Source: §3.3.1 · p. 12

3.3.2 Preference Fine-tuning

  • Preference fine-tuning follows SFT with chosen-versus-rejected response triplets; DPO was supplemented with chosen-response SFT loss to limit drift from preference data.The protocol also used high-quality chosen responses selected by Nemotron-4-340B-Reward when ground truth was unavailable.Source: §3.3.2 · pp. 12–13
  • Three RPO iterations produced the final Nemotron-4-340B-Instruct checkpoint, with improvement continuing across iterations and one iteration already improving all tasks.RPO approximates reward gaps using the policy’s implicit rewards, addressing DPO’s tendency to overfit and unlearn high-quality rejected responses.Source: §3.3.2 · p. 13

3.4.1 Automatic Benchmarks

  • Nemotron-4-340B-Instruct is competitive with available open-access instruct models, with zero-shot evaluation emphasized as the realistic instruction-following setting.Table 5 covers automatic benchmarks; the authors prioritize zero-shot evaluation because it tests instruction following without examples and resembles real-world interaction.Source: §3.4.1 · p. 14, Table 5 · p. 14
  • RPO raised MT-Bench from 7.90 to 8.22 and IFEval Prompt-Strict-Acc from 61.7 to 79.9.Across stages, CodeSFT improved HumanEval from 57.3 to 70.7; General SFT improved other categories with a slight HumanEval decline, DPO raised most metrics with a slight MT-Bench drop, and RPO boosted all metrics.Source: Table 6 · p. 15, §3.4.1 · p. 15

3.4.2 Human Evaluation

  • Annotators rated 136 prompts across 10 task categories on a six-point scale, using randomized responses and guidelines centered on helpfulness and truthfulness.The scale had five quality levels plus a failure-to-follow-instructions level; annotators also assessed perceived response length to separate verbosity preferences from instruction-following ability.Source: §3.4.2 · pp. 15–16
  • Against GPT-4-1106-preview, Nemotron-4-340B-Instruct achieved an overall win:tie:loss ratio of 28.19%:46.57%:25.24%, with comparable or better win rates except on extraction and rewrite.Results were strong on multi-turn chat.Source: §3.4.2 · p. 16, Figure 5 · p. 16
  • 79.41% of Nemotron-4-340B-Instruct responses were judged an appropriate length versus 74.02% for GPT-4-1106-preview, mainly due to fewer long or verbose responses.Long or verbose responses were 20.10% versus 25.74%, respectively.Source: Table 7 · p. 16, §3.4.2 · p. 17

3.4.3 Safety Evaluations

  • Safety evaluation included AEGIS, Garak vulnerability scanning, and human content red teaming; Garak scans across multiple categories yielded nominal-to-good performance.AEGIS uses a safety taxonomy and evaluates prompts and dialogues with safety classifiers.Source: §3.4.3 · pp. 17–18
  • Nemotron-4-340B-Instruct had a very low unsafe-response rate and was comparable to Llama-3-70B-Instruct, though some minor unsafe responses involved criminal planning and regulated substances.AEGIS assessments found negligible unsafe responses in violence, suicide and self-harm, sexual-minor, PII, harassment, threat, and Needs Caution categories.Source: §3.4.3 · p. 17, Figure 6 · p. 18
  • Nemotron-4 340B Chat remained vulnerable to malware requests and adversarial hallucinations, and its jailbreak pass rate was below 30%.The study did not assess whether generated malware code worked; it also found direct incorrect answers to impossible logic problems and false denials of prime numbers.Source: §3.4.3 · pp. 18–19

4 Conclusion

The conclusion presents three Nemotron-4 340B models as open-access releases, alongside training and inference code and a synthetic data generation pipeline. The authors expect these resources to support further LLM and AI application development.

Model family

  • The released family comprises Nemotron-4-340B-Base, Nemotron-4-340B-Instruct, and Nemotron-4-340B-Reward.The models are provided under a permissive open-access license.Source: §4 · p. 19

Open resources

  • The authors release training and inference code and provide details about their synthetic data generation pipeline.They report illustrating the pipeline’s effectiveness.Source: §4 · p. 19

Intended impact

  • The authors believe these models will stimulate further development of LLMs and AI applications.Source: §4 · p. 19

A Examples of Incapable Tasks

Table 8 is described as giving examples of tasks that an LLM cannot perform.

Table scope

  • Table 8 presents examples of tasks described as incapable of being performed by the LLM itself.Source: Table 8 · p. 26

Examples

  • The table’s stated scope is examples of tasks, rather than a reported benchmark comparison.Source: Table 8 · p. 26

What the caption establishes

  • The caption does not specify the tasks or provide results for them.Source: Table 8 · p. 26

B Prompts Used for Synthetic Prompt Generation

The appendix provides prompt templates for creating varied synthetic topics, questions, writing tasks, document-based instructions, and math or coding problems.

Generate Macro Topics; Generate Subtopics based on Macro Topics

Generate Math Macro Topics; Generate Math Subtopics based on Macro Topics

Classify if an entity is related to Math; Classify if an entity is related to Python Programming

Generate Python Macro Topics; Generate Python Subtopics based on Macro Topics

Generate Open Q&A questions based on Topics; Revise Open Q&A questions

Generate Writing task based on Topics and Document Types; Revise Writing tasks

Generate Instructions based on the Given Document

B.5 Math&Coding

  • Math prompts generate varied problems, while Python coding prompts request diverse problems for beginner, medium, and advanced programmers.Source: §B.5 · pp. 29–30

C Prompts Used for Eliciting User Turns in Synthetic Dialogue Generation

The appendix shows templates for generating user turns from conversation history, with variants for ordinary, complex, or concise follow-ups. It also includes an LLM-as-judge prompt for comparing assistant responses.

Prompt V1: Normal User Turn

Prompt V2: Complex User Turn

Prompt V3: Concise User Turn

D Prompts used in LLM-as-Judge

E Prompt Template for Evaluations

The evaluation appendix gives prompt templates, including HumanEval and MBPP templates following OpenCodeInterpreter and prompts for math and question answering.

E Prompt Template for Evaluations

F Full Evaluations on Topic-Following

On the topic-following benchmark with human-annotated distractors, Nemotron-4-340B-Instruct achieves the best overall accuracy among the evaluated models.

F Full Evaluations on Topic-Following

  • 81.7% F1 on distractor identification: Nemotron-4-340B-Instruct outperforms the other evaluated models by a large margin.Table 9 reports precision, recall, and F1 for distractor and on-topic categories across GPT-4-1106-preview, Mixtral-8x22B-Instruct-v0.1, Llama-3-70B-Instruct, and Nemotron-4-340B-Instruct.Source: F Full Evaluations on Topic-Following · p. 32, Table 9 · p. 32
  • Nemotron-4-340B-Instruct achieves the best overall accuracy, with high precision, recall, and F1 in both distractor and on-topic categories.Source: F Full Evaluations on Topic-Following · p. 32

G Human Evaluation Dataset Distribution

The appendix presents the human-evaluation dataset's distribution across categories.

G Human Evaluation Dataset Distribution

  • Table 10 presents the human-evaluation dataset distribution across categories.Source: Table 10 · p. 32

H Safety Taxonomy

The safety taxonomy defines categories covering hate, sexual content, violence, self-harm, threats, and sexual contexts involving minors.

H Safety Taxonomy

  • Hate covers content that expresses, incites, or promotes prejudice toward protected groups, including slurs, but excludes neutral identity references and contextualized hate speech.Source: H Safety Taxonomy · p. 33
  • Sexual content covers explicit or implicit sexual behavior and material meant to arouse or promote sexual services, excluding non-erotic or contextualized material such as medical education.Source: H Safety Taxonomy · p. 33
  • Violence includes intentional physical force or power causing or likely to cause harm, threats of such force, and vivid descriptions, but excludes contextualized violence and self-harm.Source: H Safety Taxonomy · p. 33
  • Self-harm includes interest, support, confessions, intent, or planning involving self-harm or suicide, except contextualized discussion.Source: H Safety Taxonomy · p. 33
  • Threats include communications of intent to cause harm or loss and intimidation used for coercion or control.Source: H Safety Taxonomy · p. 33
  • The taxonomy includes sexual conduct involving minors, such as encouraging sex acts, sharing sexual media, or arranging in-person sexual activity.Source: H Safety Taxonomy · p. 33
Loading 2406.11704v2…