PaperlayerView exact revision on arXiv

Source-linked AI summary

Aligner: Efficient Alignment by Learning to Correct

Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, Tianyi Qiu, Yaodong Yang

arXiv:2402.02416v5cs.CLcs.AIcs.LG

TL;DR

Existing alignment methods can be resource-intensive and difficult to adjust as requirements change. Aligner learns answer corrections with a small plug-in model; across 11 upstream LLMs, it improves helpfulness and harmlessness, with an average helpfulness enhancement of 68.9%.

  • Problem

    Existing alignment methods face high training-resource consumption and difficulty ensuring consistent performance as real-world alignment requirements change.

    Source: §1 · p. 1

  • Method

    A small model learns correctional residuals between preferred and non-preferred answers and is stacked on the upstream LLM as a plug-in module.

    Source: §1 · p. 2

  • Results

    Across 11 upstream models, Aligner-7B improves helpfulness and harmlessness, with an average helpfulness enhancement of 68.9%.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Aligner can be incorporated into multi-round RLHF/DPO pipelines to help mitigate reward collapse and error accumulation.

    Source: §4 · p. 8

    Aligner adds inference cost because it requires an extra model on top of the original model.

    Source: §6.1 · p. 10

Abstract

from arXiv · show

With the rapid development of large language models (LLMs) and ever-evolving practical requirements, finding an efficient and effective alignment method has never been more critical. However, the tension between the complexity of current alignment methods and the need for rapid iteration in deployment scenarios necessitates the development of a model-agnostic alignment approach that can operate under these constraints. In this paper, we introduce Aligner, a novel and simple alignment paradigm that learns the correctional residuals between preferred and dispreferred answers using a small model. Designed as a model-agnostic, plug-and-play module, Aligner can be directly applied to various open-source and API-based models with only one-off training, making it suitable for rapid iteration. Notably, Aligner can be applied to any powerful, large-scale upstream models. Moreover, it can even iteratively bootstrap the upstream models using corrected responses as synthetic human preference data, breaking through the model's performance ceiling. Our experiments demonstrate performance improvements by deploying the same Aligner model across 11 different LLMs, evaluated on the 3H dimensions (helpfulness, harmlessness, and honesty). Specifically, Aligner-7B has achieved an average improvement of 68.9% in helpfulness and 23.8% in harmlessness across the tested LLMs while also effectively reducing hallucination. In the Alpaca-Eval leaderboard, stacking Aligner-2B on GPT-4 Turbo improved its LC Win Rate from 55.0% to 58.3%, surpassing GPT-4 Omni's 57.5% Win Rate (community report).

1 Introduction

Aligner is presented as a lightweight, model-agnostic alternative for alignment needs that change faster than resource-intensive methods can address. It learns to correct upstream answers while preserving the upstream model, and the paper reports improvements across multiple models.

1 Introduction

  • Existing alignment methods face high training-resource consumption and difficulty ensuring consistent performance as real-world alignment requirements change.The paper notes that undesirable behavior outside alignment training can be difficult to address immediately with time-consuming SFT and RLHF.Source: §1 · p. 1
  • Aligner learns correctional residuals between preferred and non-preferred responses, then stacks on an upstream model without requiring its parameters.Unlike RLHF, it uses an extra module rather than training and loading multiple models; resource demand depends on the desired Aligner efficacy, not upstream model size.Source: §1 · p. 2
  • Aligner’s early layers assess original-answer quality to set the degree of reference and correction, while middle and late layers implement that decision.The paper describes this residual behavior as simpler than directly mapping queries to aligned answers.Source: §1 · p. 2
  • For aligning a 70B LLM, Aligner-7B uses 11.25 times fewer training parameters than DPO and 22.5 times fewer than RLHF2.The paper attributes this efficiency to avoiding extra actor, critic, reward, and reference models.Source: §1 · p. 2
  • 17.5% helpfulness and 26.9% harmlessness improvements on GPT-4 are reported for Aligner-7B, among results spanning 11 upstream models.The method is described as plug-and-play across API-based and open-source, safety-aligned and safety-unaligned models.Source: §1 · p. 2

2 Aligner

Aligner is trained to transform an upstream model’s initial answer into a corrected answer, using preference data and a residual-correction strategy. Its training resource requirements remain constant as the upstream model scales, and the module can be deployed after one training session.

2 Aligner

  • After one training session on preference data, the Aligner can be deployed with different upstream models.The paper describes these integrations as zero-shot use of the trained Aligner.Source: §2 · p. 3, Table 1 · p. 4
  • Aligner is a conditional seq2seq model that takes a query and preliminary answer and generates a corrected, aligned answer.It improves an existing answer rather than directly learning how to answer the query, reducing the model-capacity requirement.Source: §2 · p. 3
  • Residual Correction first warm-ups an identity Aligner on Q-A-A data, then trains it on Q-A-C data.The strategy learns semantic correctional residuals between original answers and corrections.Source: §2 · p. 3
  • For a 70B source model, DPO needs 11.25 times and RLHF 22.5 times more training resources than Aligner; Aligner’s requirements remain constant as source-model scale increases.For a 7B source model, DPO requires 1.125 times and RLHF 2.25 times more resources than Aligner.Source: §2 · pp. 3–4

3 Experiments

The experiments test Aligner across diverse datasets and upstream models, then examine its performance, ablations, and learned correction behavior. Aligner-7B improves helpfulness and harmlessness across 11 models, while ablations and interpretability experiments probe its training and correction mechanisms.

3.1 Experiment Setup

  • The setup combined HH-RLHF, PKU-SafeRLHF, and a 50K preference dataset, then evaluated three Aligner sizes across five datasets and API-based and open-source models on the 3H dimensions.Source: §3.1 · pp. 4–5

3.2 Experiment Results

  • Figure 2 shows Aligner-7B improving GPT-4 over-refusals and both scores for unaligned Alpaca-7B, while raising Llama2-70B-Chat helpfulness without reducing harmlessness.Source: Figure 2 · p. 5
  • Aligner-7B improved all 11 upstream models after one training session, averaging 21.9% higher helpfulness and 23.8% higher harmlessness across models.Source: §3.2 · pp. 5–6
  • Across five datasets, Aligner enhanced upstream models across parameter scales; TruthfulQA showed no extra hallucination information, while reasoning gains rose with Aligner size.Source: §3.2 · p. 6

3.3 Ablation Study

  • Identity-mapping warm-up improved BeaverTails and HarmfulQA performance, peaking with 10k to 50k examples, though selecting the proportion is challenging and resource-intensive.Source: §3.3 · p. 6, Figure 3 · p. 6
  • Aligner surpasses CAI and Self-Critique in helpfulness and harmlessness, while prompt-based baselines typically require repeated iterations, longer inference, and more context.Source: §3.3 · p. 6, Table 2 · p. 7
  • Across HH-RLHF, PKU-SafeRLHF, and Ultra-Feedback, Aligner matched or exceeded SFT, RLHF, and DPO, with results consistent across preference and correction datasets.Source: §3.3 · p. 7, Table 3 · p. 7

3.4 Interpretability Experiments

  • Representation control shifted correction magnitude from copying to stronger correction; LAT indicates early layers select correction degree and middle and late layers implement it.Source: §3.4 · p. 7, Figure 4 · p. 8

4 Multi-round RLHF training via Aligner

Aligner can refine upstream-model responses into synthetic preference data for repeated RLHF or DPO. The authors report that this multi-round approach improves safety while increasing helpfulness across rounds.

4 Multi-round RLHF training via Aligner

  • Aligner is proposed to mitigate reward collapse from preference data that diverges from the upstream model’s answer distribution, with errors accumulating across rounds.The passage also notes a risk of reward over-optimization, such as producing longer responses regardless of safety.Source: §4 · p. 8
  • Aligner refines an upstream response into A∗, and the resulting (Q, A, A∗) pairs can train the next RLHF or DPO round.Repeated rounds can form a synthetic preference dataset for further upstream-model training.Source: §4 · p. 8, Figure 5 · p. 8
  • By bringing synthetic preference data closer to the upstream model’s answer distribution, Aligner is described as mitigating out-of-distribution reward-model collapse and providing repeatable preference-data generation.The method is also described as inheriting the shift from dispreferred to preferred responses.Source: §4 · pp. 8–9
  • Aligner-corrected preference data improved safety and increased helpfulness monotonically across rounds, unlike typical multi-round RLHF/DPO, which improved utility but left responses unsafe.The experiments used three rounds on Alpaca2-7B.Source: §4 · p. 9

5 Related Work

Related work covers multi-model RLHF, inference-time methods, and self-refinement. Aligner is presented as a model-agnostic module that can refine outputs from other models after one training session.

5 Related Work

  • Unlike RLHF’s distributed training of multiple models and reliance on human expert annotation, Aligner uses a constrained seq2seq model and does not depend on upstream model parameters.This makes it suitable for API-based models without parameter access.Source: §5 · p. 9
  • Many inference-time methods are sensitive to the upstream model or require logits, while Aligner is described as a seamless, single-training-session module for 11 upstream model types.The passage contrasts Aligner with methods that refine prompts or select among upstream responses.Source: §5 · p. 9
  • Unlike self-refinement methods that depend on one model improving its own outputs, Aligner uses a separate model to refine outputs from larger models, including GPT-4.The authors say this avoids computation spent on self-critiquing; combining Aligner with an external critique model is proposed as future work.Source: §5 · p. 9

6 Conclusion

Aligner is presented as an efficient, lightweight, model-agnostic alignment approach with reported gains across helpfulness and harmlessness. The authors note added inference costs and warn that the released dataset could be misused for harmful purposes.

6 Conclusion

  • Aligner is presented as an efficient, lightweight, model-agnostic approach that improves computational efficiency without additional components such as actor, critic, or reward models.Source: §6 · p. 10
  • Aligner-7B averages 68.9% helpfulness and 23.8% harmlessness gains; Aligner-2B raises GPT-4 Turbo LC Win Rate from 55.0% to 58.3%, above GPT-4 Omni’s 57.5%.Source: §6 · p. 10

6.1 Limitations and Future Work

  • Using an external Aligner adds inference costs; the authors suggest smaller Aligners and streamlined corrections as possible ways to reduce the burden.Future work also includes multi-turn dialogue and domain-specific alignment.Source: §6.1 · p. 10

6.2 Ethics and Impact

  • The authors plan to release the Aligner dataset under CC BY-NC 4.0 and warn that it could theoretically be used to train assistants for harmful or malicious purposes.Source: §6.2 · p. 10

A Application: Weak-to-Strong Correction via Aligner

The application uses a weak Aligner to correct a strong model’s responses and generate fine-tuning labels. The authors report gains from SFT across the Llama2 series, while RLHF and DPO show trade-offs in helpfulness and safety.

A Application: Weak-to-Strong Correction via Aligner

  • The application addresses the difficulty of providing reliable feedback as advanced AI systems undertake increasingly complex tasks that are hard for humans to grasp.The cited discussion frames this as the Superalignment problem and relates it to weak-model supervision of strong models.Source: A Application: Weak-to-Strong… · p. 15
  • Weak-to-Strong Correction uses a weak Aligner to correct a strong upstream model’s response and generate labels for fine-tuning that model.The authors describe a trade-off between mimicking weaker labels and using the strong model’s reasoning, and say their approach balances that tension.Source: A Application: Weak-to-Strong… · p. 15, Figure 8 · p. 15
  • Aligner-7B and Aligner-13B labels improve Llama2-series strong models in every reported SFT scenario.Table 4 covers weak-to-strong correction on 7B, 13B, and 70B upstream models using existing alignment methods; the supplied passages do not give cell values.Source: Table 4 · p. 16, A Application: Weak-to-Strong… · p. 16
  • For weak-to-strong training, each dataset example is a query, an Alpaca-7B answer, and an Aligner-7B correction; SFT uses the correction as its label, while RLHF and DPO treat it as preferred to the original.Source: A Application: Weak-to-Strong… · p. 16
  • RLHF and DPO improve some metrics but do not fully retain the strong model’s original capabilities, particularly helpfulness, which the authors associate with conservative, less-informative responses.The authors suggest reward modeling and policy optimization may introduce feature noise and information loss compared with SFT’s direct label mapping.Source: A Application: Weak-to-Strong… · p. 16

B.1 Evaluation Datasets

The appendix describes the evaluation datasets, preference-rate calculation, and human review process used to assess Aligner.

B.1 Evaluation Datasets

  • Empathetic Dialogue prompts cover seven emotion categories, with 4,300 training items and 1,300 test items.The categories include angry, confident, embarrassed, proud, sad, lonely, terrified, and devastated.Source: §B.1 · p. 17
  • Evaluation spans dialogue summarization, safety, factuality, coding, academic knowledge, mathematics, and instruction-following benchmarks.The datasets include DialogSum, BeaverTails, HarmfulQA, TruthfulQA, HumanEval, MMLU, MATH, and MT-Bench.Source: §B.1 · p. 17

B.2 Evaluation Calculation Methods

  • GPT-4 and crowdsourced annotators judge original and corrected answers, with win, draw, and loss counts used to calculate preference rates.The passage defines ω as the success rate and Nw, Ne, and Nl as correctional-answer wins, draws, and losses.Source: §B.2 · p. 18

B.3 GPT-4 and Human Evaluation

  • The evaluation process combines GPT-4 judgments with human review, using crowdsourced annotators and fixed quality-control staff.Quality-control personnel worked with the team to clarify assessment requirements and revise the process.Source: §B.3 · p. 18
  • The project reports 28 full-time crowdsourced workers, average hourly wages of USD 8.02–9.07, and an institutional ethics review.The stated wage range exceeds the cited Beijing minimum wage of USD 3.69; the Peking University committee audited the dataset’s use.Source: §B.3 · p. 18

B.4 Consistency and Reliability of Evaluation Methods.

The appendix reports agreement rates among evaluation groups and points to prompt and hyperparameter details for self-refinement baselines.

B.4 Consistency and Reliability of Evaluation Methods.

  • Agreement was 85% between quality inspectors and the team, 82% between inspectors and crowdsourced annotators, and 80% between annotators and the team.The checks sampled 100, 200, 300, and 500 items.Source: §B.4 · p. 18

B.5 Hyper-Parameters of Self-Refine/Self-Critique Methods

  • The self-refine and self-critique comparison uses the CAI prompt and reports that Aligner outperforms the baseline across different models.Hyperparameters for Aligner and the baseline models are listed in Table 7.Source: §B.5 · p. 18
  • The appendix provides separate GPT-4 evaluation prompts for harmlessness and helpfulness.The harmlessness prompt focuses on minimizing toxicity and bias; the helpfulness prompt asks evaluators to disregard safety-related elements.Source: Table 5 · p. 19, Table 6 · p. 19

B.6 Details of Interpretability Methods

The appendix details how Aligner’s correction behavior is probed, then reports its inference-time trade-off and performance across domains, intervention baselines, and prompt feedback.

B.6 Details of Interpretability Methods

  • The interpretability analysis forms copy and correction activation sets, derives a representation vector with PCA, and tests its effect by adding it to the residual stream.The controlled corrections are compared with original responses using Levenshtein Ratio.Source: §B.6 · p. 20

C.1 The Discussion of Limitation: Inference Time

  • Aligner’s inference time is roughly consistent with same-sized Llama2-Chat models, while Llama2-7B-Chat + Aligner outperforms Llama2-13B-Chat.Llama2-13B-Chat + Aligner is slightly inferior to Llama2-70B-Chat; the authors describe smaller models with Aligner as alternatives under limited resources.Source: §C.1 · p. 21, Table 8 · p. 21

C.2.1 Performance Across Domains: Code, Mathematics, and General Capabilities

  • The authors report zero-shot generalization on code, mathematics, instruction-following, and general-capability benchmarks, attributing it to base-model generalization and copy-correction behavior.The copy-correction approach can favor copying in some out-of-domain question-answer scenarios.Source: §C.2.1 · p. 21, Table 9 · p. 22

C.2.2 Aligner vs. Inference-time Intervention Methods

  • In supplementary experiments using best-of-N (BoN) and beam search as alternative inference-time intervention methods, Aligner continued to demonstrate superior performance compared with both approaches.Source: §C.2.2 · p. 21

C.2.3 Feedback Intervention during Aligner Refinement Process

  • Prompt feedback during refinement improves metrics on BeaverTails and empathy evaluations while retaining instruction-following capability.The prompts ask Aligner to prioritize empathy, helpfulness, or harmlessness; the authors describe this as enabling fine-grained adjustments after training.Source: §C.2.3 · p. 22

C.2.4 Length Bias Analysis

The appendix checks whether response length may bias evaluations and gives examples of prompt-guided and model-specific corrections.

C.2.4 Length Bias Analysis

  • Not all Aligner-corrected responses are longer, so response length alone is not necessarily decisive in evaluation outcomes.The authors also describe a double-blind human comparison of original and corrected responses; the values are presented in Table 13.Source: §C.2.4 · p. 22, Table 13 · p. 24

C.3 Examples on Aligner

  • Prompt guidance can steer Aligner toward precise adjustments, including an Instruct-Aligner setup.The paper’s examples describe safer corrections to dangerous answers from unaligned models and added risk information when GPT-4 refuses to answer directly.Source: §C.2.4 · p. 22, §C.3 · p. 23

D Aligner Implementation Details and Hyperparameters

The appendix details how Aligner’s correction data was assembled and annotated, then summarizes its training setup and implementation. The pipeline combines human and model annotations, while training uses multiple model sizes and data volumes.

D Aligner Implementation Details and Hyperparameters

D.1 The Details of Query-Answer Dataset

  • After deduplication, 27K queries yielded 57K quality-filtered query-answer pairs for correction annotation.The queries came from sources including Stanford Alpaca, ShareGPT, and HH-RLHF.Source: §D.1 · p. 24

D.2.1 Human Annotation Details

  • Human annotators handled approximately 27.10% of correction answers, separating harmlessness annotation from helpfulness correction-triple annotation.For harmlessness, annotators wrote a harmless response; for helpfulness, they produced a correction using the original pair and harmless answer.Source: §D.2.1 · p. 24

D.2.2 GPT-4 Annotation Details

  • GPT-4 annotated approximately 43.19% of correction answers, with its annotation prompt provided in Table 18.The appendix also includes example original and corrected answers for emotional queries and red-team prompts.Source: Table 14 · p. 25, Table 15 · p. 26, §D.2.2 · p. 27

D.2.3 Llama2-70B-Chat Annotation Details

  • Llama2-70B-Chat annotated approximately 6.69% of correction answers, and its prompt appears in Table 19.Figure 9 describes the correction-data process: a 50K Q-A-C dataset was built from 27K queries, with corrections annotated by GPT-4, human annotators, and Llama2-70B-Chat, followed by quality filtering.Source: §D.2.3 · p. 27, Table 19 · p. 28, Figure 9 · p. 28

D.3 The Training Code of Aligner vs. SFT

  • The Aligner training pseudocode requires only a one-line modification to an SFT code base.The correction template asks for editing a question-answer pair to make it more helpful and harmless.Source: §D.3 · p. 28, §D.4 · p. 30

D.4 Hyper-Parameters for the Aligner Training

  • The appendix provides separate tables for Aligner and baseline training hyperparameters.Table 20 covers Aligner training at different sizes, while Table 21 covers baseline methods.Source: §D.4 · p. 29, Table 20 · p. 29, Table 21 · p. 31
Loading 2402.02416v5…