PaperlayerView exact revision on arXiv

Source-linked AI summary

Hermes 3 Technical Report

Ryan Teknium, Jeffrey Quesnelle, Chen Guang

arXiv:2408.11857v1cs.CL

TL;DR

Wide-ranging language-model capabilities can be difficult for users to control, while instruct tuning offers a way to steer models through imperative prompts. Hermes 3 combines instruct and tool use with training that emphasizes faithful, neutral prompt following; it achieves state-of-the-art performance among open-weight models on several public benchmarks.

  • Problem

    Wide-ranging language-model capabilities can be difficult for average users to control.

    Source: §1 · p. 1

  • Method

    Hermes 3 is an instruct and tool-use model trained to follow system and instruction prompts faithfully and neutrally.

    Source: §1 · p. 1

  • Results

    Hermes 3 achieves state-of-the-art performance among open-weight models on several public benchmarks.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Hermes 3 maintains coherent, contextually relevant multi-turn conversations, especially in role-play with Llama 3.1’s 128K-token context window.

    Source: §2 · p. 2

    Training the 405B model at an 8K-token context length required at least seven HGX nodes with CPU parameter offloading, which reduced training efficiency by 45%.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Instruct (or "chat") tuned models have become the primary way in which most people interact with large language models. As opposed to "base" or "foundation" models, instruct-tuned models are optimized to respond to imperative statements. We present Hermes 3, a neutrally-aligned generalist instruct and tool use model with strong reasoning and creative abilities. Its largest version, Hermes 3 405B, achieves state of the art performance among open weight models on several public benchmarks.

1 Introduction

Broad language-model capabilities can be difficult for average users to control. Hermes 3 is presented as an instruct and tool-use model that emphasizes steerability and neutral instruction following, with state-of-the-art performance among open-weight models on several public benchmarks.

1 Introduction

  • Wide-ranging language-model capabilities can be difficult for average users to control.The introduction describes base models trained across diverse text as difficult to control.Source: §1 · p. 1
  • Instruct tuning trains a base model to respond to imperative statements, while chat tuning steers it toward a helpful-assistant persona.System prompts can further guide how instructions are interpreted, and tool use can add computation or data retrieval.Source: §1 · p. 1
  • Hermes 3 is an instruct and tool-use model with strong reasoning and creative abilities, achieving state-of-the-art performance among open-weight models on several public benchmarks.Source: §1 · p. 1
  • Its training spans synthetic reasoning tasks and expressive applications, and strongly encourages faithful, neutral responses to system and instruction prompts.The report contrasts this training emphasis with popular closed-weight commercial models that may refuse instructions on moral grounds.Source: §1 · p. 1
  • The authors argue that guardrails and active intervention belong at the system level, given language models’ limited direct agency.They caution that intervening at the model level can limit potential lines of thinking.Source: §1 · p. 1

2 Model Overview

Hermes 3 is a fine-tuned Llama 3.1 model whose behavior is highly sensitive to its system prompt. The report describes strong multi-turn interaction and additional judgment, reasoning, coding, tool-use, and retrieval capabilities.

2 Model Overview

  • Hermes 3 is fine-tuned from Llama 3.1 8B, 70B, and 405B models and is highly sensitive to its system prompt.The 405B model may not adopt a helpful-assistant persona when the system prompt is empty.Source: §2 · p. 2
  • Hermes 3 maintains coherent, contextually relevant multi-turn conversations, especially in role-play using Llama 3.1’s 128K-token context window.In role-play, it adopts and consistently maintains personas while adapting language, knowledge base, and behavior.Source: §2 · p. 2

2.1 Extended capabilities

  • Hermes 3 shows significant improvements in judgment and reward modeling, with potential use in fine-tuning and iterative model improvement.Figure 7 gives an example of multi-turn evaluation.Source: §2.1 · p. 2
  • Its agentic capabilities include structured outputs, scratchpads, internal monologues, diagrams, and step-labeled reasoning and planning.The report says these features collectively improve handling complex tasks, explaining approaches, and communicating ideas.Source: §2.1 · p. 2
  • Hermes 3 is described as capable of generating functional code and explanations, and of combining planning with external tools and retrieved information.Tools use the Hermes Function Calling standard, while retrieval sources can be cited using the <co> tag.Source: §2.1 · p. 2

3 Data Mixture

The Hermes 3 dataset combines selected existing instruction sources with generated domain-specific data, then filters the resulting conversations for quality. The final mixture contains approximately 390 million tokens, with response tokens contributing to the training loss.

3 Data Mixture

  • Data curation combines selected existing sources with generated domain-specific instructions to cover code, mathematics, role-play, agentic tasks, and other areas.General Instructions account for a substantial portion of their 236 million tokens; generation schemes were inspired by Evol-Instruct [34].Source: §3 · p. 3
  • Filtering removes refusals, improperly formatted responses, missing or empty turns, and conversations outside token-length thresholds.The process also prioritizes conversations generated by the strongest models.Source: §3 · p. 3
  • Approximately 390 million tokens make up the final mixture: 270 million (69%) are response tokens contributing to cross-entropy loss, and 120 million are instruction tokens.Table 1 presents proportions and token counts by dataset category.Source: §3 · p. 3, Table 1 · p. 3

4 Training Recipe

Hermes 3 is trained with supervised fine-tuning followed by direct preference optimization. DPO gives a moderate benchmark benefit for the 8B model, while larger models retain their SFT checkpoints.

4 Training Recipe

  • The training recipe has two phases: supervised fine-tuning (SFT), followed by direct preference optimization (DPO).Source: §4 · p. 3

4.1 Supervised Fine-Tuning

  • SFT uses Llama 3.1 models at 8B, 70B, and 405B parameters, with a 128K-token context length.The 8B and 70B runs use a 7 × 10−6 peak learning rate; the 405B run uses 3.5 × 10−6.Source: §4.1 · pp. 3, 5
  • SFT packs samples with variable-length Flash Attention to prevent cross-sample contamination, achieving 96% packing efficiency.Instruction and tool-output tokens are ignored in the loss; the target sequence length is 8192 tokens.Source: §4.1 · p. 4, Figure 3 · p. 4
  • SFT checkpoints are selected using a min-max normalized average of public benchmarks, including GPT4All, AGIEval, IFEval, and MT-Bench.The passage describes per-epoch scores for the 70B model, but supplies no values here.Source: §4.1 · p. 4, Table 2 · p. 5
  • For 405B SFT at an 8K context, the minimum stated setup is seven HGX nodes with CPU offloading, estimated to reduce training efficiency by 45%.The authors say higher-dimensional parallelism is likely necessary for future 405B runs to avoid overly large batch sizes.Source: §4.1 · p. 5

4.2 DPO

  • DPO uses LoRA adapters to avoid keeping both reference and trained models in GPU memory, with r = 32, α = 16, and dropout 0.05.The reported DPO setup also uses a 3 × 10−6 peak learning rate and NEFTune with α = 5.Source: §4.2 · pp. 5–6
  • DPO moderately improves 8B benchmarks, but yields negligible gains for larger models, so their SFT checkpoints are retained.Table 4 covers post-DPO benchmarks for the 8B model.Source: §4.2 · p. 6, Table 4 · p. 6

5 Evaluations

Final downstream evaluations combine benchmarks used during SFT with tasks from the Open LLM Leaderboard. The 405B models are evaluated under FP8 quantization.

5 Evaluations

  • Final evaluations include BBH, MATH, GPQA, MuSR, MMLU, and MMLU-PRO alongside evaluations used in SFT.Source: §5 · p. 7
  • The 405B models are evaluated using FP8 round-to-nearest weight quantization, channelwise activations, and per-token scales.The passage names llm-compressor for vLLM as the quantization tooling.Source: §5 · p. 7
  • Table 5 presents final downstream task evaluations.Source: Table 5 · p. 7

A Generation Samples

The generation samples include examples of empty-system-prompt output, reward modeling, retrieval-augmented generation, and agentic coding. The included prompt examples show evaluation, document-grounded answering, and a structured code-problem response format.

A Generation Samples

Loading 2408.11857v1…