Source-linked AI summary
SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF
Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xianchao Wu, Oleksii Kuchaiev
TL;DR
RLHF’s single-dimensional reward feedback leaves users unable to adjust individual response attributes at inference. STEERLM adds attribute-conditioned SFT, and its 43B model outperformed all baselines in both automatic and human evaluations.
Problem
Method
STEERLM extends SFT with annotated reward attributes, predicting attributes for prompt-response data and conditioning generation on those attributes alongside the prompt.
Source: §1 · p. 2, §3.3 · p. 3Results
STEERLM 43B outperformed all baseline models on both automatic and human evaluations.
Source: §4.5 · p. 7Takeaways & Limitations
Runtime attribute controls can serve game and red-teaming use cases, though malicious users might exploit toxic or violent generation.
Source: Ethics Statement · p. 9Evaluations are English-only, so whether the positive results extend to other languages remains untested.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Model alignment with human preferences is an essential step in making Large Language Models (LLMs) helpful and consistent with human values. It typically consists of supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) stages. However, RLHF faces inherent limitations stemming from a complex training setup and its tendency to align the model with implicit values that end users cannot control at run-time. Moreover, reward models in RLHF stage commonly rely on single-dimensional feedback as opposed to explicit, multifaceted signals that indicate attributes such as helpfulness, humor, and toxicity. To address these limitations, we propose SteerLM, a supervised fine-tuning method that empowers end-users to control responses during inference. SteerLM conditions responses to conform to an explicitly defined multi-dimensional set of attributes, thereby empowering a steerable AI capable of generating helpful and high-quality responses while maintaining customizability. Experiments show that SteerLM trained on open source datasets generates responses that are preferred by human and automatic evaluators to many state-of-the-art baselines trained with RLHF while being much easier to train. Try SteerLM at https://huggingface.co/nvidia/SteerLM-llama2-13B
1 Introduction
RLHF can improve alignment but adds training complexity and offers limited user control over individual response attributes. STEERLM conditions generation on explicit attributes, and the paper reports that its 43B model outperforms state-of-the-art baselines on the Vicuna benchmark.
1 Introduction
- RLHF training is complex and its monodimensional preferences limit users’ ability to adjust response attributes at inference.The introduction contrasts this with SFT alone, which does not distinguish high- from low-quality responses.Source: §1 · pp. 1–2
- STEERLM trains a predictor to annotate response attributes, then conditions generation on the prompt and those attributes.The paper highlights flexible inference-time customization, including attributes such as humor and toxicity.Source: Figure 1 · p. 1, §1 · p. 2
- STEERLM 43B outperforms state-of-the-art baselines, including ChatGPT-3.5, on the Vicuna benchmark.The paper presents STEERLM as a simpler alignment alternative that uses the language-modeling objective.Source: §1 · p. 2
2 Related Work
Related work describes SFT and RLHF alignment approaches, while identifying limitations in training complexity and single-dimensional preference signals. Prior attribute grounding often targets narrow tasks and a single attribute, unlike STEERLM’s stated focus on general open-domain conversations and multiple attributes.
2 Related Work
- Prior SFT work used model-generated, crowdsourced, proprietary, or bootstrapped datasets; some open-source datasets underperformed commercial RLHF models.The cited examples include Zhang et al. (2023), Peng et al. (2023), Dettmers et al. (2023), and Köpf et al. (2023b).Source: §2 · p. 2
- RLHF improves instruction following and downstream tasks, but its complex training setup remains a hurdle to widespread adoption.Related approaches include offline RLHF, conditional sequence modeling, direct preference optimization, and response ranking.Source: §2 · p. 2
- Single-dimensional reward signals cannot represent varied objectives such as helpfulness, humor, and toxicity, which can vary across domains.The passage notes that low scores on some attributes may be relevant in cases such as toxic gaming NPCs.Source: §2 · pp. 2–3
- Earlier attribute grounding focused on narrowly defined dialogue tasks with one attribute, whereas STEERLM targets open-domain conversations with multiple attributes.The described tasks include code assistance, poetry writing, and planning.Source: §2 · p. 3
3 SteerLM
STEERLM uses supervised fine-tuning with explicit attribute values to align and steer language-model responses. Its four-step process predicts attributes, annotates datasets, trains attribute-conditioned generation, and bootstraps training with scored model-generated responses.
3 SteerLM
- STEERLM is presented as a simple alignment approach trained solely with the language-modeling objective and as a computationally efficient alternative to RLHF.The method is organized into four steps.Source: §3 · p. 3
3.1 Step 1. Attribute Prediction Model
- The Attribute Prediction Model learns to predict response attributes such as quality, toxicity, and humor from prompts and responses.Open Assistant examples represent attributes as integer values, and the attributes are the target output when conditioning on prompt x and response y.Source: §3.1 · p. 3
3.2 Step 2. Annotating Datasets using Attribute Prediction Model
- The predictor annotates prompt-response datasets, extending attribute labels to other datasets and helping address annotation noise and calibration differences.The resulting annotated dataset is used in the next training step.Source: §3.2 · p. 3
3.3 Step 3. Attribute Conditioned SFT
- Attribute-conditioned SFT trains responses from prompts and attribute labels, learning from both high- and low-quality responses through offline data.This avoids RLHF’s online response generation and evaluation setup, simplifying training configuration.Source: §3.3 · pp. 3–4
3.4 Step 4. Bootstrapping with High Quality Samples
- For bootstrapping, STEERLM samples diverse responses using quality-9 attribute combinations, scores them with the predictor, and uses the labeled samples for another SFT round.The process pairs training prompts with high-quality attribute settings, then predicts attributes for generated responses.Source: §3.4 · p. 4
4 Experiments
The experiments train SteerLM on OpenAssistant data and evaluate it against instruction-following baselines using automatic and human comparisons. SteerLM 43B outperforms the baselines on both evaluations, with the authors noting that longer, more diverse responses may contribute to its automatic-evaluation advantage.
4.1 Training Datasets
- OASST contributes 48.8K unfiltered conversations and seven selected attributes, while HH-RLHF contributes 348.6K conversations annotated by the Attribute Prediction model.The selected OASST attributes are quality, toxicity, violence, helpfulness, creativity, humor, and inappropriateness; 910 M-SID prompt-response pairs are also included.Source: §4.1 · p. 5
4.2 Base Models for STEERLM
- SteerLM 43B uses a 43B base model trained on 1.1 trillion tokens, while SteerLM 13B uses the Llama 2 13B base model.The 43B base model was trained on multilingual web, news, books, scientific publications, and code; it is used for both Attribute Prediction and Attribute Conditioned SFT.Source: §4.2 · p. 5
4.3 Training details
- Both models are trained for five epochs with a global batch size of 128 and maximum sequence length of 4096 tokens.Training uses 16 A100-DGX nodes, each with eight A100-80GB GPUs, and Adam with a 5e-6 learning rate and 1e-2 weight decay.Source: §4.3 · p. 5
4.4 Evaluation
- SteerLM is evaluated against ChatGPT 3.5, text-davinci-003, Guanaco 65B, Vicuna 13B, and OASST LLaMA 30B SFT and RLHF models.GPT-4 scores competing responses for 80 Vicuna prompts; human evaluation uses 12 university-educated annotators and a subset of three top automatic-evaluation models.Source: §4.4 · pp. 5–6
4.5 Results
- SteerLM 43B outperforms all baselines on both automatic and human evaluations, with 74 Elo points over the closest baseline automatically and 59 on human evaluation.Its responses average 1906 characters and 144 unique words, versus ChatGPT 3.5’s 1193 characters and 77 unique words; GPT-4 may favor longer, more diverse responses.Source: §4.5 · p. 7
- Compared with Guanaco 65B, SteerLM 43B performs better despite its smaller base model and fewer training tokens; the authors attribute this likely to using all OASST data, including lower-quality conversations.Guanaco uses only the highest-quality response at each conversation-tree level.Source: §4.5 · p. 7
5 Ablation Study
The ablations assess attribute labels, data selection, predicted annotations, dataset augmentation, and bootstrapping. Attribute labels yield the largest reported gain, while each of the other tested changes also improves performance.
Addition of attribute labels
- 16.5% performance increase is reported after adding attribute labels, with the quality attribute identified as the primary contributor.The ablation starts from fine-tuning on the full OASST dataset without attribute information.Source: §5 · pp. 7–8
Finetuning on only High Quality Data
- Up to 1.9% performance improvement is reported when Attribute Conditioned SFT uses 3,400 OASST samples with the highest human-annotated quality.This tests training on a small high-quality subset instead of the broader dataset.Source: §5 · p. 8
Predicted attributes
- 4.6% performance benefit is reported for using Attribute Prediction model outputs rather than human annotations in SteerLM 43B training.The authors suggest this helps calibrate quality annotations across samples and mitigate noise in crowdsourced labels.Source: §5 · p. 8
Augmentation with Anthropic HH-RLHF
- 1.0% additional performance increase is reported after augmenting OpenAssistant data with Anthropic HH-RLHF.The authors report that the Attribute Prediction model can annotate high-quality data from datasets beyond its training dataset.Source: §5 · p. 8
Bootstrapping with High-Quality Samples
- 0.5% performance gain is reported from bootstrapping with high-quality samples.The authors hypothesize that sampling explores the response space and adds higher-quality data.Source: §5 · p. 8
6 Steerability demonstration
The steerability demonstrations vary toxicity and humor attributes at inference. They report that toxicity can be increased and that humor settings alter the style of responses to different prompts.
6.1 Toxicity
- At the default toxicity setting, SteerLM 43B is slightly less toxic than ChatGPT; increasing the setting allows more toxic outputs.The authors identify NPC dialogue and red-teaming as potential applications of this control.Source: §6.1 · p. 8
6.2 Humor
- For a day-planning prompt, raising humor from 0–3 to 6 adds a cheeky remark, while setting it to 9 produces a cheesy remark.At humor 0–3, both SteerLM 43B and ChatGPT 3.5 provide an extensive activity list.Source: §6.2 · p. 9
7 Conclusion
STEERLM aligns language models through supervised fine-tuning and lets users adjust response attributes at inference without retraining. It achieves state-of-the-art Vicuna benchmark results and is preferred over the compared models in human evaluation.
7 Conclusion
- Users can adjust the model’s value system at inference without retraining, including humor level and toxicity tolerance.The conclusion presents this as a customizable approach to model alignment.Source: §7 · p. 9
- STEERLM trains both its attribute prediction model and language model using supervised fine-tuning.The authors describe the resulting training process as easy to implement and straightforward compared with RLHF.Source: §7 · p. 9
- STEERLM achieves state-of-the-art results on the Vicuna benchmark and is preferred over the compared models in human evaluation.The conclusion reports both benchmark results and human-evaluation preferences.Source: §7 · p. 9
Limitations
The authors identify higher GPU-hour and energy costs than parameter-efficient fine-tuning, and English-only evaluation as limitations.
Limitations
- Both STEERLM models are fully supervised fine-tuned, making them relatively costly in GPU hours and energy compared with LoRA and prompt tuning.The limitation applies to both the Attribute Prediction Model and Attribute-Conditioned SFT models.Source: Limitations · p. 9
- Automatic and manual evaluations use an English-only benchmark, leaving performance on multilingual benchmarks untested.The authors call for multilingual evaluation to test whether the positive results extend to other languages.Source: Limitations · p. 9
- The authors say multilingual benchmarks are needed to test whether the positive results extend to evaluations in other languages.This is a stated evaluation need, not a reported multilingual result.Source: Limitations · p. 9
Ethics Statement
STEERLM allows users to request toxic or violent responses at runtime, which can serve some use cases but may also be exploited. The authors suggest restricting runtime controls to selected attributes unless users demonstrate an acceptable use case.
Ethics Statement
- Runtime controls can enable toxic or violent responses, which the authors identify as a feature for games or red-teaming.The passage gives situations where characters may need to respond differently.Source: Ethics Statement · p. 9
- The authors warn that malicious users might exploit runtime control of toxic or violent responses outside intended use cases.Source: Ethics Statement · p. 9
- One proposed safeguard is to expose only a subset of attributes at runtime unless users demonstrate an acceptable use case.The authors state that developers can control which attributes users may adjust.Source: Ethics Statement · p. 9
A.1 Attribute Prediction Model prompt templates
The appendix gives prompt-template details for attribute prediction and attribute-conditioned SFT, describes data-collection requirements, and reports a model-size comparison.
A.1 Attribute Prediction Model prompt templates
- The attribute-prediction template appends encoded value attributes after the assistant response, and the language-model loss is calculated on the encoded values.The template uses special tokens to mark messages and the encoded attributes.Source: §A.1 · p. 12
A.2 Attribute Conditioned SFT prompt templates
- The Attribute-Conditioned SFT template places encoded attributes before the assistant response and calculates language-model loss on the response.Source: §A.2 · p. 12
A.4 Effect of model size on performance
- SteerLM outperforms Guanaco at every comparable model size.The comparison uses SteerLM 8B and 22B against Guanaco 7B, 13B, and 30B; Figure 3 presents the model-size results.Source: Figure 3 · p. 12, §A.4 · p. 12
A.3 Further Data Collection
- SteerLM data collection annotates each prompt-response sample with selected attributes, while RLHF collects pairwise comparisons among multiple responses.The appendix contrasts SteerLM’s per-response ratings with RLHF’s comparisons among k=4–9 responses per prompt.Source: §A.3 · p. 12
- SteerLM’s one-time, absolute-scale response ratings make its dataset annotation cost approximately an order of magnitude lower than RLHF.The appendix states that RLHF may require 36 annotations per prompt and iterative data collection as the actor improves.Source: §A.3 · p. 12