HelpSteer3: Human-Annotated Feedback and Edit Data to Empower Inference-Time Scaling in Open-Ended General-Domain Tasks

1 Introduction

• Detailed feedback can explain why parts of a response need work and how to improve them, beyond a simple helpfulness judgment.The introduction contrasts this with feedback based on preferences or fixed dimensions such as factual correctness, creativity, complexity, and coherence.Source: §1 · p. 1
• Fixed feedback dimensions may not capture holistic response quality or identify where errors occur, especially in long responses.The usefulness of natural-language feedback without ground-truth answers remains unclear for general-domain tasks.Source: §1 · pp. 1–2
• Many inference-time scaling techniques rely on verifiable answers, restricting their application to tasks such as math, multiple-choice questions, and competitive coding.The cited approaches include verifier-based reinforcement learning, process reward models, generative verifiers, search, self-consistency, and execution feedback.Source: §1 · p. 2
• Open-ended challenges such as research ideation and personal writing may lack verifiable answers, motivating complementary approaches to inference-time scaling.The introduction notes that GPT-4o without thinking was preferred over o1 for personal writing.Source: §1 · p. 2

Efficacy of LLMs in providing feedback and making edits to their own responses is unclear

• Whether models can productively edit general-domain responses using feedback, and how this compares with editing without feedback, remains under-explored.The cited literature includes reports of self-correction in other settings, but the authors knew of no study of feedback-conditioned editing for general-domain tasks.Source: Efficacy of LLMs in providing feedback… · p. 2

Our Solution: HelpSteer3 Enables Training of Dedicated Feedback-Edit Models that are Inference-Time Scalable for SOTA performance

• HelpSteer3 contains feedback and edits from more than 7,000 annotators across 80-plus regions for diverse open-ended general-domain tasks.Annotators provide textual helpfulness feedback and use it to edit original responses.Source: Our Solution: HelpSteer3 Enables… · p. 2
• Dedicated Feedback and Edit models trained on HelpSteer3 can improve responses to open-ended general-domain tasks at inference time.The system is scaled by generating more initial responses, feedback, and edits.Source: Our Solution: HelpSteer3 Enables… · p. 2
• 92.7 on Arena Hard was achieved by an optimally scaled Feedback-Edit setup based on Llama 3 family 70B models, exceeding OpenAI o1-preview-2024-09-12 at 90.4 and DeepSeek R1 at 92.3.The paper also reports improvements on MT Bench and AlpacaEval.Source: Our Solution: HelpSteer3 Enables… · p. 2

2.1 Data Collection

• Prompts span General, STEM, Coding, and Multilingual categories, with unsafe or personally identifying tasks filtered or skipped.Coding and Multilingual prompts come from ShareGPT, while General and STEM prompts come from WildChat; multiple models generate responses, including intermediate turns for multi-turn prompts.Source: §2.1 · p. 3
• Annotators provide 2–10 sentences of English feedback on response helpfulness, and separate annotators edit responses using consolidated feedback.Three to five annotators assess each task; feedback is consolidated using the three annotators whose helpfulness judgments agree most, and perfectly helpful or unhelpful responses are excluded from editing.Source: §2.1 · pp. 3–4

2.2 Data Pre-processing

• Feedback Demonstration retains three agreeing feedback items per prompt, while Edit Demonstration uses change summaries to identify feedback addressed by edits and includes feedback-order permutations.Feedback with no improvement suggestions and feedback not mentioned in the change summary are excluded from the edit demonstrations.Source: §2.2 · p. 4
• Edit Preference pairs good edits with bad edits or unchanged responses, but this dataset covers only General and STEM tasks.Bad edits are edits not based on the provided feedback; unchanged responses copy the original response.Source: §2.2 · p. 4

2.3 Data Analysis

• More than half of both demonstration datasets comes from specialist annotators in STEM, Coding, and Multilingual tasks, and the data spans 14 programming and 13 natural languages.The authors describe this mix as supporting training across domains.Source: §2.3 · p. 5
• Feedback averages 437.8 characters overall, with multilingual feedback shorter at 339.2 and coding feedback longer at 518.6 characters.The authors associate shorter multilingual feedback with straightforward prompts and longer coding feedback with checking implementations and identifying concrete issues.Source: §2.3 · p. 5
• Good edits change response length by an average of 32.5 characters, while bad edits decrease it by 122.1 characters on average.Original responses average 1,642.7 characters; edited responses in General, STEM, and Multilingual stay within 100 characters of originals, while Coding edits increase by 782 characters on average.Source: §2.3 · p. 5

3.1 Evaluation

• AlpacaEval 2.0 Length Controlled, MT Bench, and Arena Hard measure easy, medium, and hardest prompts, respectively.All responses are generated greedily; MT Bench response length is reported and MT Bench also serves as a checkpoint-validation metric.Source: §3.1 · p. 5
• Each initial response receives 10 sampled feedback items; after filtering perfectly helpful judgments, up to three are selected to condition greedy edit generation.Feedback is sampled at temperature 0.7 and top p 0.9, balancing feedback-generation cost against having most responses retain three feedback items after filtering.Source: §3.1 · pp. 5–6

3.2 Model Training

• Feedback and Edit supervised fine-tuning starts from Llama 3.3 70B Instruct, with the Edit reward and reinforcement-learning models initialized from the Edit SFT model.Datasets are split 95% for training and 5% for validation; Feedback SFT trains for one epoch and selects the step-1,400 checkpoint, while Edit SFT selects the step-385 checkpoint.Source: §3.2 · p. 6
• The Edit reward model uses Bradley–Terry training on paired edits, with one batch containing both bad-versus-good and unchanged-versus-good comparisons.Training stops at one epoch because the authors note that Bradley–Terry reward models can overfit beyond that; the selected checkpoint is step 80.Source: §3.2 · p. 6
• Edit reinforcement learning applies REINFORCE Leave One Out to the Edit SFT model, guided by the Edit reward model.The selected checkpoint is step 45, based on validation performance.Source: §3.2 · p. 6

• Feedback + Edit substantially improves MT Bench, AlpacaEval, and Arena Hard for both Llama-3.1-Nemotron-70B-Instruct and Llama-3.3-70B-Instruct.Source: Table 3 · p. 6, §4 · p. 6
• Llama-3.1-Nemotron-70B-Instruct was included as the strongest open-source 70B model by Arena Hard, AlpacaEval 2.0 LC, and MT Bench.Source: §4 · p. 6
• Both models also gain AlpacaEval 2 Length Controlled, suggesting improvements persist after controlling for response length.Source: §4 · p. 6

Self-Feedback and Self-Edit

• Self-feedback/edit raises AlpacaEval (57.6 to 64.6) and MT Bench (8.98 to 9.11) but lowers Arena Hard (85.0 to 84.6) versus the initial model.The setup also produces substantially longer responses: 3587.0 characters versus 2199.8 for the initial model.Source: Table 4 · p. 6, §5 · p. 7

Feedback and Self-Edit

• Prompted Self-Edit raises AlpacaEval (62.8 to 66.2) but lowers MT Bench (9.16 to 8.94) and Arena Hard (87.0 to 85.4) versus Feedback + Edit.Source: Table 4 · p. 6, §5 · p. 7

Removing RL from Edit Training

• Adding RL stops the SFT-only Edit's ~30% unchanged-response behavior; MT Bench rises 9.12 to 9.16 and Arena Hard 86.4 to 87.0, while AlpacaEval falls 64.4 to 62.8.Source: Table 4 · p. 6, §5 · p. 7

Importance of Feedback

• Feedback-free Edit changes AlpacaEval from 64.4 to 67.4, but Arena Hard from 86.4 to 84.5; RL without feedback lacks a feedback-relevance training signal.MT Bench is similar (9.12 vs. 9.14); feedback-free Edit also scores below the initial model on Arena Hard (84.5 vs. 85.0).Source: Table 4 · p. 6, §5 · p. 7

• Feedback is advantageous on challenging Arena Hard tasks, where meaningful improvements require more targeted feedback.Source: §5 · p. 8

Initial Responses per Prompt

• Scaling from 2 to 16 initial responses per prompt increases Arena Hard from 89.0 to 90.2, suggesting Feedback-Edit complements Best-of-N sampling.Each initial response is processed by Feedback-Edit, and the highest-reward edited response is selected.Source: §6 · p. 8

Effective Feedback per Response

• Scaling from 2 to 16 effective feedback increases Arena Hard from 87.0 to 89.0.The setup generates 16, 32, or 64 feedback items and reranks them by constructive-criticism keywords to obtain 4, 8, or 16 effective feedback items.Source: §6 · p. 8

Edited Responses per Feedback set

• Scaling from 2 to 16 edits per feedback set increases Arena Hard from 88.4 to 89.6.For each prompt, original response, and feedback set, the system generates edits and selects the highest-reward one.Source: §6 · p. 8

Scaling multiple axes simultaneously

• The combined scaling experiment increases responses per prompt and effective feedback per response, reaching up to 8 responses and 16 effective feedback under compute constraints.This configuration generates about 16 times as many tokens as default greedy generation, similar to Best-of-N 16.Source: §6 · p. 8
• 92.7 Arena Hard with Llama-3.3-Nemotron-70B-Select surpasses OpenAI o1-preview-2024-09-12 at 90.4 and DeepSeek R1 at 92.3.The result follows experiments using Llama-3.1-Nemotron-70B-Reward, which reached 91.0 and outperformed Best-of-N 16 at 88.5.Source: §6 · p. 9

Human Evaluation

• In a 50-prompt human evaluation, Feedback-Edit responses had 16 improvements versus 7 degradations among 44 valid comparisons.Twenty-one were rated about the same; factuality, coherence, and conciseness were the main improvement justifications.Source: §6 · p. 9

Data

• The training data uses 22,644 unique first-turn prompts, split into 21,850 training and 794 validation prompts.First-turn prompts are used to minimize generation prefix lengths for throughput; vLLM is used to increase generation throughput.Source: §7 · p. 9

Model Training

• SFT runs for one epoch on Llama 3.1 8B Instruct, Llama 3.3 70B Instruct, and Llama-3.1-Nemotron-70B-Instruct initializations.The selected checkpoints are at steps 120, 150, and 160, respectively.Source: §7 · p. 9

Results

• Llama-3.3-70B-Instruct improves from 35.0 to 61.6 on AlpacaEval and from 62.4 to 88.8 on Arena Hard after distillation.Table 5 covers distillation with various Instruct models; higher is better for all metrics except Length.Source: §7 · p. 9, Table 5 · p. 9
• Distillation improves all initial models across MT-Bench, AlpacaEval, and Arena Hard, generally by less than optimally applied inference-time Feedback-Edit.The authors suggest the generated data may help train models for latency-sensitive applications, while greater benefits remain available from inference-time scaling.Source: §7 · p. 9

Conclusion

• Dedicated Feedback and Edit models are presented as useful for inference-time scaling in open-ended general-domain tasks.The conclusion reports the benchmark outcome alongside this claim.Source: §8 · p. 9
• 92.7 on Arena Hard surpasses OpenAI o1-preview-2024-09-12 at 90.4 and DeepSeek R1 at 92.3.The paper also reports improvements on MT-Bench and AlpacaEval.Source: §8 · p. 9

Societal Impact

• The authors describe potential advantages in customization, ease of deployment, latency, and personalization.These are presented as possible additional advantages of Feedback-Edit inference-time scaling.Source: Ethical Considerations · p. 10
• Disaggregated serving of Feedback and Edit models can avoid requiring both models to share one compute resource.The passage contrasts this with monolithic models such as DeepSeek R1, described as having over 600B parameters.Source: Ethical Considerations · p. 10
• Sampling initial responses, feedback, and edits in parallel gives the system about twice greedy-generation latency, while thinking tokens must be generated sequentially.The authors note sequential thinking can produce higher latency upper bounds, especially when a model generates many more thinking tokens than response tokens.Source: Ethical Considerations · p. 10
• Users may select model-generated feedback aligned with their preferences and use it to ground edits in a controllable way.The authors describe this as opening research avenues for personalized alignment.Source: Ethical Considerations · p. 10

Ethical Considerations

• Annotation practices include local-minimum-wage pay, measures against overwork, health and safety protections, clear contracts, and channels to appeal platform decisions.The passage also describes measures to reduce unpaid job-search time and mitigate discrimination risks.Source: Ethical Considerations · p. 10

Limitations

• The current feedback-sampling, filtering, and re-ranking approach is not compute-optimal.Constrained decoding or other logit manipulation may improve its compute efficiency.Source: Limitations · p. 11
• The prompt collection may not represent the most complex queries: it excludes long prompts and responses, and queries requiring knowledge after July 2024.The sources are ShareGPT prompts collected before April 2023 and WildChat prompts collected before April 2024.Source: Limitations · p. 11
• Feedback-Edit is presented as complementary to other inference-time-scaling methods, whose usefulness may depend on the setting.Source: Limitations · p. 11

A Prompt Preprocessing

• Prompt preprocessing filters out non-English, code-response, very short, and identity-related prompts, then classifies topics and upsamples prompts scored 4 or 5 for complexity.The topic classification uses BERTopic; complexity is predicted by prompting Nemotron 4 340B Instruct.Source: A Prompt Preprocessing · p. 17

C Bad Edit Details

• Feedback and edit annotators sometimes disagreed; inspection found feedback annotators more often correct, possibly because editors face greater effort for major edits.Source: C Bad Edit Details · p. 18

D Further Dataset Statistics

• The Coding and Multilingual subsets cover 14 programming and 13 natural languages, though a few languages are over-represented.Source: D Further Dataset Statistics · p. 18, Table 6 · p. 18
• Table 7 reports annotator geographic locations by dataset in ISO-3166 alpha-2 order; 0.0 denotes a share below 0.05%.Source: Table 7 · p. 19

E Illustrative Example

• The bad edit adds vendor pricing and legal-planning advice beyond the feedback’s requested changes.Source: E Illustrative Example · p. 20
• In the tourism-marriage example, the good edit removes the “unique experiences” item while retaining the other feedback-aligned content.The feedback describes that item as generic and unnecessary.Source: E Illustrative Example · p. 20

F Aligned Model Evaluation Details

• MT Bench reports mean judge scores across 160 prompts, AlpacaEval 2.0 LC measures length-corrected preference over GPT-4-1106-turbo, and Arena Hard reports win rate over GPT-4-0314.Arena Hard uses 500 challenging first-turn instructions; more than half concern coding problems.Source: F Aligned Model Evaluation Details · p. 21

G Training Details

• Training used Nemo-Aligner, a 4096-token maximum sequence length, AdamW, and tuned learning rates; Edit RL used a 5e-7 learning rate and 0.01 KL.Data longer than 4096 tokens, less than 3% of each dataset, was discarded; rollouts used four responses per prompt, up to 2048 tokens, temperature 1, and top p 0.9.Source: G Training Details · p. 21

H Compute requirements

• Table 8 reports training compute in H100 node-hours for experiments run on nodes with eight H100-80GB SXM GPUs.Source: Table 8 · p. 21

Note: Below is one single feedback

• The example feedback identifies fictional citations, shallow analysis, missing search methodology, and absent practical or research implications as weaknesses in a literature-review response.The accompanying examples range from broadly positive assessments to suggestions for concision and more coverage of implementation challenges.Source: Note: Below is one single feedback · p. 22, J Example Feedback by Feedback Model · p. 22

K Additional Comparisons

• Chain-of-Thought prompting maintains MT Bench performance but substantially degrades AlpacaEval 2 LC and Arena Hard performance relative to the base Llama-3.1-Nemotron-70B-Instruct model.Source: K Additional Comparisons · p. 23
• CritiqueLLM feedback with Self-Edit raises AlpacaEval from 57.6 to 66.0 but lowers MT Bench from 8.98 to 7.96 and Arena Hard from 85.0 to 82.2.Using the paper’s Feedback model with Self-Edit scores higher than CritiqueLLM with Self-Edit on each benchmark.Source: K Additional Comparisons · p. 23, Table 9 · p. 23
• On Gemma 2 27B Instruct, Feedback and Edit substantially improve MT Bench and Arena Hard while maintaining AlpacaEval within standard error.The comparison applies the Feedback and Edit system to a non-Llama model.Source: K Additional Comparisons · p. 23, Table 9 · p. 23
• 70B Feedback and Edit models improve results across nearly all metrics for 8B and 70B starting models, whereas 8B Feedback and Edit models generally fail to improve both and often degrade performance.The stated exceptions are AlpacaEval 2 LC for the 8B starting model with 70B feedback/edit models, which is maintained, and AlpacaEval 2 LC for the 70B starting model with 8B models, which improves less than the 70B no-RL baseline.Source: K Additional Comparisons · p. 23, Table 9 · p. 23