PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment

Geyang Guo, Ranchi Zhao, Tianyi Tang, Wayne Xin Zhao, Ji-Rong Wen

arXiv:2311.04072v2cs.CL

TL;DR

SFT and RLHF offer different trade-offs for aligning language models, while existing non-RL methods largely lack fine-grained signals about response quality. FIGA pairs initial and revised responses and uses token-level quality signals; the authors report that it outperforms the initial supervised-finetuned model and PPO.

  • Problem

    SFT mainly imitates demonstrations, while prior non-RL approaches largely use response-level quality signals that do not show what makes a response good or bad.

    Source: §1 · pp. 1–2

  • Method

    FIGA pairs initial responses with revised responses and uses a fine-grained loss to weight tokens representing good or bad actions.

    Source: §1 · p. 2

  • Results

    FIGA outperforms the initial supervised-finetuned model by 3.2 points and the strong PPO method by 1.8 points, and surpasses all baselines across benchmarks.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    FIGA retains SFT’s efficiency and ease of implementation while better understanding and learning correct alignment behaviors.

    Source: §5 · p. 9

    The dataset-construction approach assumes a reward model for assessing alignment is available.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Alignment with human preference is a desired property of large language models (LLMs). Currently, the main alignment approach is based on reinforcement learning from human feedback (RLHF). Despite the effectiveness of RLHF, it is intricate to implement and train, thus recent studies explore how to develop alternative alignment approaches based on supervised fine-tuning (SFT). A major limitation of SFT is that it essentially does imitation learning, which cannot fully understand what are the expected behaviors. To address this issue, we propose an improved alignment approach named FIGA. Different from prior methods, we incorporate fine-grained (i.e., token or phrase level) quality signals that are derived by contrasting good and bad responses. Our approach has made two major contributions. Firstly, we curate a refined alignment dataset that pairs initial responses and the corresponding revised ones. Secondly, we devise a new loss function can leverage fine-grained quality signals to instruct the learning of LLMs for alignment. Extensive experiments have demonstrated the effectiveness of our approaches by comparing a number of competitive baselines.

1 INTRODUCTION

The introduction frames alignment as a trade-off between SFT’s imitation-based simplicity and RLHF’s complexity, then presents FIGA’s fine-grained signals and revised-response dataset as an alternative.

1 INTRODUCTION

  • SFT is easy to implement and can boost task solving, but imitates demonstrations and depends on their quality; RLHF explores semantic space but is complicated and can suffer reward collapse.The introduction also notes that SFT may face large distribution shifts between model outputs and imitated outputs.Source: §1 · p. 1
  • Prior approaches without RL largely use response-level quality labels, which indicate whether a response is good or bad but not what makes it so.The authors identify this as a limitation for learning alignment behaviors.Source: §1 · p. 2
  • FIGA contrasts low-quality initial responses with high-quality revisions to identify added good actions and removed or substituted bad actions.Its SPA dataset pairs initial and revised responses, and its loss assigns token-level weights to guide alignment.Source: §1 · p. 2
  • FIGA outperforms the initial supervised-finetuned model by 3.2 points and the strong PPO method by 1.8 points.The authors report promising performance in aligning language models with human preferences.Source: §1 · p. 2

2 RELATED WORK

Related work covers RLHF’s multi-stage reward-optimization pipeline and approaches that avoid reinforcement learning, including data-focused and alternative-optimization methods.

2 RELATED WORK

  • RLHF commonly uses SFT, trains a reward model from human preferences, and then applies PPO to maximize reward.The cited discussion notes complex coding and hyperparameter selection as challenges, as well as high memory use from loading three to four models.Source: §2 · p. 2
  • Non-RL alignment work either constructs preference-aligned supervised datasets or explores substitutes for PPO, including preference-enriched training data, preference-based losses, and controllable text generation.The passage describes these as different approaches to alignment without reinforcement learning.Source: §2 · pp. 2–3
  • The preference information used by these methods is at sentence level and lacks more fine-grained supervision signals.Source: §2 · p. 3

3 APPROACH

FIGA pairs weak initial responses with minimally revised, higher-quality versions, then uses their differences as fine-grained alignment signals. Its training objective emphasizes desirable edits and discourages undesirable ones while retaining SFT-like simplicity and an RL-like focus on behavior quality.

3.1 CURATED ALIGNMENT DATASET

  • A reward model filters for weak initial responses, strong demonstrations, and a clear reward gap before examples enter SPA.The method assumes an available reward model to assess alignment and set filtering thresholds.Source: §3.1 · pp. 3–4
  • SPA pairs queries with weak initial responses and minimally revised versions that improve quality while preserving the aligned model’s linguistic style and semantic distribution.The revisions are guided by human-preferred demonstrations, and both responses are retained to derive fine-grained signals.Source: §3.1 · pp. 3–4

3.2 FINE-GRAINED QUALITY-AWARE ALIGNMENT TUNING

  • Levenshtein-based token comparison rewards added or substituted tokens in revised responses and penalizes deleted or substituted tokens in initial responses.Tokens unchanged by revision are assigned a weight that can be set to be ignored.Source: §3.2 · pp. 4–5
  • FIGA trains on fine-grained contrasts between initial and revised responses, emphasizing consequential words or phrases rather than treating every token as equally important.The approach is presented as easy to implement like SFT while capturing revision effects in a way similar to RLHF.Source: §3 · p. 3, §3.2 · p. 5

3.3 DISCUSSION

  • FIGA’s token weights act as a simplified advantage function, avoiding a learned critic and previous rollout model used in the RL comparison.The paper characterizes FIGA as a simplified, more efficient version of RL with a similar objective.Source: §3.3 · pp. 5–6

4 EXPERIMENT

The experiments compare FIGA with established alignment methods and test its dataset and weighting choices. FIGA surpasses the baselines across benchmarks, including PPO trained with four times as much data.

4.1.1 BASELINE METHODS

  • The study compares FIGA with SFT, PPO, CoH, RRHF, and DPO, which use different fine-tuning, reward-optimization, ranking, or preference-comparison objectives.The methods are described as baseline approaches for evaluating FIGA.Source: §4.1.1 · p. 6

4.1.2 IMPLEMENTATION DETAILS

  • SPA contains 17,333 instances; Alpaca-7b generates initial responses, gpt-3.5-turbo revises them, and a DeBERTa reward model filters data using thresholds η1 = 1, η2 = 3, and η3 = 3.5.FIGA uses α = 1, β = 0.5, and γ = 0; bad tokens are retained only when their negative log-likelihood is below 0.6.Source: §4.1.2 · pp. 6–7

4.1.3 EVALUATION TASKS

  • Evaluation uses a 3,608-entry test set scored by reward model, plus out-of-distribution benchmarks for knowledge, alignment, and open-ended generation.The named benchmarks include MMLU, WinoGender, CrowS-Pairs, TruthfulQA, Vicuna, and WizardLM.Source: §4.1.3 · p. 7

4.2 EXPERIMENTAL RESULTS

  • FIGA surpasses all baselines across benchmarks, including PPO trained with four times the data.The comparison is reported in Table 2.Source: Table 2 · p. 7, §4.2 · p. 7
  • Figure 3 compares FIGA and baseline response win rates on Vicuna and WizardLM.Vicuna is shown on the left and WizardLM on the right; the passage does not state the figure’s outcomes.Source: §4.2 · p. 7, Figure 3 · p. 8

4.3.1 PERFORMANCE COMPARISON W.R.T. SUBPAR ALIGNMENT DATASET

  • Reward-score filtering yields +0.21 reward score and +2.8 benchmark points, while training with revisions yields +3.8 points on average.The experiments also report strong FIGA performance on typical instruction data unseen by the reward model.Source: §4.3.1 · p. 8, Table 3 · p. 8, Table 4 · p. 8

4.3.2 PERFORMANCE COMPARISON W.R.T. WEIGHTING FUNCTIONS

  • Levenshtein token selection exceeds statistical and ChatGPT annotation methods by over +1.5 and +2.6 average scores; further bad-token filtering adds +6.8.The authors also report +0.7 from penalizing undesirable actions, a 1.9 decrease when γ is non-zero, and attribute weak reward-score weighting to inaccuracies, especially out of distribution.Source: Table 5 · p. 9, §4.3.2 · p. 9

5 CONCLUSION

FIGA aligns language models by using fine-grained quality signals during fine-tuning. Its dataset pairs initial and revised responses, and the authors report higher scores than the initial SFT model and PPO.

5 CONCLUSION

  • FIGA pairs initial responses with revised responses on queries a language model cannot perform well on, then contrasts them to use fine-grained quality signals in its learning objective.The approach is presented as retaining SFT’s efficiency and ease of implementation while better learning correct alignment behaviors.Source: §5 · p. 9
  • The authors report that FIGA outperforms the initial supervised-finetuned model by 3.2 points and PPO by 1.8 points across extensive tasks.Source: §5 · p. 9
  • The conclusion presents FIGA as an approach that can better understand and learn correct alignment behaviors while retaining SFT’s efficiency and ease of implementation.Source: §5 · p. 9

A APPENDIX: DATA SOURCES

The appendix describes five datasets used as sources for the alignment data, spanning preference data, conversations, synthetic instructions, and model completions.

A APPENDIX: DATA SOURCES

  • HH-RLHF contains human preference data about helpfulness and harmlessness, as well as human-annotated red-teaming dialogues.The preference data is used to train reward models.Source: A APPENDIX: DATA SOURCES · p. 13
  • ShareGPT contains conversations with user prompts and ChatGPT responses collected through the API.Source: A APPENDIX: DATA SOURCES · p. 13
  • Synthetic Instruct GPT-J Pairwise explores model-generated outputs for synthetic instruction-oriented prompts.Source: A APPENDIX: DATA SOURCES · p. 13
  • Stanford SHP provides 385K human preferences across disciplines to distinguish relative response helpfulness.Its content is written by humans, complementing other datasets.Source: A APPENDIX: DATA SOURCES · p. 13
  • OpenOrca extends the FLAN Collection with GPT-4 and GPT-3.5 completions; this study uses its English instruction subset.Source: A APPENDIX: DATA SOURCES · p. 13

B APPENDIX: PROMPTS USED FOR DATA AUGMENTATION

The prompts guide ChatGPT to analyze why an original response is weaker, revise it minimally toward a preferred response, and score words by their contribution to quality.

B APPENDIX: PROMPTS USED FOR DATA AUGMENTATION

C APPENDIX: EVALUATION DETAILS

The appendix details benchmark-specific evaluation procedures, using likelihood-based accuracy, perplexity-based bias rates, and ChatGPT ratings across the benchmark suite.

C APPENDIX: EVALUATION DETAILS

D APPENDIX: SPA DATASET DETAILS

The appendix estimates SPA dataset API costs and reports a substantial performance boost when FIGA is trained with 8,000 samples.

D APPENDIX: SPA DATASET DETAILS

E APPENDIX: HYPTER-PARAMETER SETTINGS

The appendix compares FIGA performance across hyper-parameter configurations and reports preferred settings for γ, α, and β.

E APPENDIX: HYPTER-PARAMETER SETTINGS

F APPENDIX: HUMAN EVALUATION

Human evaluations found that FIGA outperformed all baseline methods in pairwise quality comparisons, while errors of all assessed types decreased after FIGA training.

F APPENDIX: HUMAN EVALUATION

  • The human evaluation assessed both error types before and after training and pairwise response quality against baselines.The error assessment used 20 test instructions; pairwise comparisons used 30 Vicuna benchmark instructions.Source: F APPENDIX: HUMAN EVALUATION · pp. 15–16
  • After FIGA training, all four assessed error types significantly decreased across responses to 20 test-set instructions.Three participants assessed whether responses before and after training exhibited the errors.Source: F APPENDIX: HUMAN EVALUATION · p. 15, Table 8 · p. 16
  • FIGA outperformed all baseline methods in pairwise quality comparisons on 30 Vicuna benchmark instructions.Three evaluators compared FIGA responses with each baseline, including ChatGPT; final judgments used the mode of their assessments.Source: F APPENDIX: HUMAN EVALUATION · p. 16, Table 9 · p. 16
Loading 2311.04072v2…