PaperlayerView exact revision on arXiv

Source-linked AI summary

RAIN: Your Language Models Can Align Themselves without Finetuning

Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, Hongyang Zhang

arXiv:2309.07124v2cs.CL

TL;DR

LLMs can diverge from human preferences, while conventional alignment relies on finetuning and alignment data. RAIN addresses this gap by combining self-evaluation with rewindable inference on frozen models, achieving stronger safety results without training or additional data. The paper reports improved harmlessness on HH and improved safety preservation across aligned models, while acknowledging longer inference time and limited attack scope.

  • Problem

    Conventional LLM alignment typically requires finetuning with preference data, motivating alignment of frozen models without additional data.

  • Method

    RAIN combines self-evaluation and rewind operations during inference to guide generation in frozen LLMs.

  • Results

    RAIN improves alignment performance across safety tasks and models without additional data or training, including 97% harmlessness for LLaMA 30B versus 82% vanilla inference on HH.

  • Takeaways & Limitations

    RAIN shows that frozen LLMs can self-align through an inference-time plug-in without external alignment data or parameter updates.

  • Takeaways & Limitations

    RAIN requires longer inference time than standard autoregressive inference, and the paper does not claim robustness against adaptive attacks.

Abstract

from arXiv · show

Large language models (LLMs) often demonstrate inconsistencies with human preferences. Previous research typically gathered human preference data and then aligned the pre-trained models using reinforcement learning or instruction tuning, a.k.a. the finetuning step. In contrast, aligning frozen LLMs without requiring alignment data is more appealing. This work explores the potential of the latter setting. We discover that by integrating self-evaluation and rewind mechanisms, unaligned LLMs can directly produce responses consistent with human preferences via self-boosting. We introduce a novel inference method, Rewindable Auto-regressive INference (RAIN), that allows pre-trained LLMs to evaluate their own generation and use the evaluation results to guide rewind and generation for AI safety. Notably, RAIN operates without the need of extra data for model alignment and abstains from any training, gradient computation, or parameter updates. Experimental results evaluated by GPT-4 and humans demonstrate the effectiveness of RAIN: on the HH dataset, RAIN improves the harmlessness rate of LLaMA 30B from 82% of vanilla inference to 97%, while maintaining the helpfulness rate. On the TruthfulQA dataset, RAIN improves the truthfulness of the already-well-aligned LLaMA-2-chat 13B model by 5%.

1 Introduction

RAIN investigates whether frozen LLMs can self-align without finetuning or alignment data. It combines self-evaluation with rewindable inference to improve safety while preserving helpfulness.

  • Motivation: Existing alignment methods require finetuning, human-annotated data, and substantial computational resources.The paper contrasts RAIN with RLHF, RLAIF, RRHF, RAFT, and DPO.
  • Approach: RAIN uses self-evaluation and rewindable generation to let frozen LLMs produce responses consistent with human values.The method omits the self-annotation and finetuning steps used by related self-alignment approaches.
  • Properties: RAIN is designed as a broadly applicable inference plug-in for existing autoregressive LLMs.The paper describes universality across language-generation tasks and straightforward integration.
  • Properties: RAIN aligns frozen models with memory usage matching vanilla autoregressive inference and without storing gradients or computational graphs.This is contrasted with RLHF, which requires additional models and training-related state.
  • Properties: RAIN is learning-free, requiring no human annotations or labeled or unlabeled alignment data.The paper frames this as evidence that alignment capacity can be accessed within fixed models during inference.

2 Related Work

Prior alignment work mainly fine-tunes models with reinforcement learning, alternative optimization objectives, or self-generated training data. RAIN differs by targeting safety alignment through inference-time search and rewind operations.

  • Alignment with reinforcement learning: Reinforcement-learning alignment has been used to fine-tune pretrained LLMs for helpfulness and harmlessness.The related work cites Claude and InstructGPT as examples.
  • Alignment without reinforcement learning: RRHF, RAFT, and DPO avoid reinforcement learning while still modifying training objectives for alignment.These methods are presented as more streamlined or stable training alternatives.
  • Alignment without reinforcement learning: Self-Instruct and Self-Alignment generate training data through in-context learning and then fine-tune models with gradient-based algorithms.Their self-generated data still enters a training pipeline.
  • Lookahead and backtracking: RAIN applies lookahead and backtracking to safety alignment rather than to prompting models over intermediate problem-solving thoughts.The paper distinguishes its mechanism and target from Yao et al. (2023).
  • Adversarial attack on LLMs: Other work studies adversarial attacks that induce harmful responses and defenses based on metrics such as perplexity.These approaches address attack detection or defense rather than RAIN’s inference-time self-alignment setting.

3 Rewindable Auto-regressive INference (RAIN)

RAIN searches over token-set trees, evaluates generated text, and rewinds generation to update search attributes. It combines exploitation, exploration, similarity-based propagation, and visit-count-guided token selection.

  • Overview: RAIN treats autoregressive generation as a token-set tree where search and rewind can modify previously generated choices.This addresses error propagation caused by fixed sequential tokens.
  • Node representation: Each node records an embedding, language-model probability, visit count, and value conditioned on its preceding context.Nodes and token sets are used interchangeably in the method’s notation.
  • Search procedure: The inner loop alternates forward exploration and backward rewinding, updating node attributes before later simulations.The outer loop uses the accumulated search information to guide final generation.
  • Search procedure: RAIN selects token sets using both value-based exploitation and exploration of under-visited branches.The regularization parameter c balances exploitation against exploration, while language-model probability prioritizes likely candidates.
  • Evaluation and updates: Self-evaluation scores completed candidate text, and the resulting value is propagated to related token sets.The evaluator assigns harmlessness scores, while similarity-based updates can affect sibling nodes subject to a threshold and discount factor.
  • Evaluation and updates: Similarity updates use contextual token-set embeddings to update off-path sibling nodes without treating them as identical.The method restricts updates to sufficiently similar siblings and discounts them with γ no greater than 1.
  • Outer loop: After repeated searches, normalized child visit counts become probabilities for the next token set, and search stops at a score threshold or iteration limit.Dynamic node addition can expand the search when existing children are both similar and low-valued.

4 Experiments

RAIN is evaluated across harm-free, adversarial, truthful, and sentiment-generation tasks using multiple open-source LLMs and GPT-4 or human-related evaluation procedures. Results show improved safety, truthfulness, sentiment control, and attack resistance, with effectiveness generally increasing with model scale but requiring additional computation.

  • Tasks and datasets: Experiments cover harm-free, adversarial harm-free, truthful, and controlled sentiment generation across LLaMA, LLaMA-2, Vicuna, Alpaca, and GPT-neo models.Harmlessness uses HH, adversarial robustness uses AdvBench with GCG, truthfulness uses TruthfulQA, and sentiment control uses IMDB.
  • Robustness: 14%, 45%, and 75% reductions in white-box attack success rates occur for Vicuna models with 7B, 13B, and 33B parameters, respectively.Transfer-attack success rates decrease by 25%, 47%, and 24% for the same model sizes.
  • Truthful and controlled generation: RAIN produces more truthful responses on TruthfulQA and improves positive-sentiment generation by 20% on LLaMA 7B.Larger sentiment improvements occur on Alpaca 7B and Vicuna 7B, while truthfulness gains also apply to the aligned LLaMA-2-chat 13B model.
  • Comparison with baselines: RAIN achieves efficacy comparable to or better than RLHF and RLAIF on LLaMA 30B without requiring their additional data.The comparison evaluates harmlessness on the HH dataset.
  • Analysis: RAIN’s self-evaluation exceeds random guessing, while its average time cost on LLaMA HH experiments is approximately four times vanilla inference.Repeated random sampling requires far more computation, including a 500-fold time increase for 500 trials, whereas RAIN has an acceptable overhead.

5 Conclusion

RAIN enables frozen LLMs to self-align during inference without additional data or training. Its main limitation is higher, though acceptable, inference time than standard autoregressive inference.

  • RAIN enables frozen LLMs to generate responses aligned with human preferences without additional data or further training.It integrates self-evaluation and rewind functionalities as an inference-time plug-in.
  • RAIN maintains or enhances the safety of already well-aligned models such as Vicuna and LLaMA-2 70B.
  • RAIN demands a longer yet acceptable inference time than standard autoregressive inference.The paper identifies transferring this overhead to a later fine-tuning phase as a possible future direction.

A.1 HH dataset

The HH examples show RAIN replacing harmful or biased answers with refusals, privacy-preserving responses, and more respectful alternatives, while vanilla inference provides unsafe content.

  • RAIN refuses to explain how to crash a plane into the Willis Tower, whereas vanilla inference supplies operational flight guidance.
  • RAIN declines to provide a private home address and redirects the user to official contact channels.
  • RAIN replaces a derogatory explanation about southerners with a statement emphasizing their diversity and immigrant heritage.
  • RAIN advises against hacking a personal computer, while vanilla inference explains how a keylogger could steal sensitive information.

A.2 AdvBench

On adversarial prompts, vanilla inference produces harmful instructions or scripts, whereas RAIN refuses assistance with bombs, viruses, and disruptive computer actions.

  • RAIN declines assistance with creating or releasing viruses intended to cause ecological damage.
  • RAIN refuses to create a script intended to crash a computer system, unlike vanilla inference.
  • RAIN refuses instructions for making a bomb from household items, contrasting with vanilla outputs that provide construction steps.

A.3 TruthfulQA

The TruthfulQA examples contrast vanilla responses containing false or supernatural claims with RAIN responses that reject the “circle of salt” premise but still produce an incorrect claim about mammals.

  • Vanilla inference attributes protective, purifying, and healing effects to a circle of salt, whereas the dataset answer says it has no particular effects.
  • RAIN states that a circle of salt has no magical or supernatural properties and identifies salt as sodium chloride.
  • RAIN answers that all mammals are vertebrates, matching the provided TruthfulQA answer but conflicting with the accompanying vanilla response.
  • The example’s supporting explanation states that mammals have backbones, while the vanilla response incorrectly identifies platypuses and echidnas as exceptions.

B Prompt Templates for Different Tasks

The appendix gives task-specific prompt templates for evaluating harmfulness, truthfulness, and sentiment in generated text. Each template supplies a generated response or review and asks the evaluator to classify it using explicit answer options or criteria.

  • B.1 HH and AdvBench: HH and AdvBench prompts ask whether the assistant’s generated response is harmful, evaluating the response solely on its content.The template presents the generated text after an instruction to determine harmfulness.
  • B.1 HH and AdvBench: HH and AdvBench provide two answer choices that reverse the harmless and harmful labels.The options distinguish harmless/harmful from harmful/harmless response classifications.
  • B.2 TruthfulQA: TruthfulQA prompts ask whether a generated answer is truthful given a question-answer pair.The evaluator is instructed to assess the answer against the question and answer context.
  • B.2 TruthfulQA: The TruthfulQA template ends by asking directly whether the answer is truthful.The closing instruction converts the preceding criteria into a binary evaluation question.
  • B.3 Controlled sentiment generation: Controlled sentiment prompts ask whether a generated movie review is positive based on whether the reviewer likes the movie.The evaluator receives the generated review after this sentiment criterion.
  • B.3 Controlled sentiment generation: Controlled sentiment generation offers positive, negative, and neutral classifications for the movie review.The options include both polarity directions and a neutral category.
Loading 2309.07124v2…