PaperlayerView exact revision on arXiv

Source-linked AI summary

The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning

Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, Yejin Choi

arXiv:2312.01552v1cs.CLcs.AI

TL;DR

The paper asks whether alignment tuning substantially changes base LLMs, given evidence that small amounts of supervised tuning can yield strong alignment. It compares token distributions between base and aligned models, then introduces URIAL, a tuning-free in-context alignment method; URIAL reduces the gap to tuned models and can match or surpass them on strong base models. The authors conclude that alignment may largely concern stylistic behavior, while noting that tuning remains necessary for some task types.

  • Problem

    Direct evidence remains limited on how alignment tuning changes base LLM behavior and whether effective alignment requires SFT or RLHF.

  • Method

    The paper analyzes token-distribution shifts between base and aligned LLMs and proposes URIAL, which uses a system prompt and a few stylistic in-context examples without tuning model weights.

  • Results

    URIAL with as few as three constant in-context examples can effectively align base LLMs and surpass SFT- or SFT+RLHF-aligned models on strong base models.

  • Takeaways & Limitations

    Most alignment changes concern a small fraction of stylistic and safety-related tokens, supporting further study of pre-training versus alignment contributions and tuning-free alignment.

  • Takeaways & Limitations

    URIAL is not suggested as a replacement for SFT or RLHF in all scenarios, including coding, mathematics, and interactive-agent tasks.

Abstract

from arXiv · show

The alignment tuning process of large language models (LLMs) typically involves instruction learning through supervised fine-tuning (SFT) and preference tuning via reinforcement learning from human feedback (RLHF). A recent study, LIMA (Zhou et al. 2023), shows that using merely 1K examples for SFT can achieve significant alignment performance as well, suggesting that the effect of alignment tuning might be "superficial." This raises questions about how exactly the alignment tuning transforms a base LLM. We analyze the effect of alignment tuning by examining the token distribution shift between base LLMs and their aligned counterpart. Our findings reveal that base LLMs and their alignment-tuned versions perform nearly identically in decoding on the majority of token positions. Most distribution shifts occur with stylistic tokens. These direct evidence strongly supports the Superficial Alignment Hypothesis suggested by LIMA. Based on these findings, we rethink the alignment of LLMs by posing the research question: how effectively can we align base LLMs without SFT or RLHF? To address this, we introduce a simple, tuning-free alignment method, URIAL. URIAL achieves effective alignment purely through in-context learning (ICL) with base LLMs, requiring as few as three constant stylistic examples and a system prompt. We conduct a fine-grained and interpretable evaluation on a diverse set of examples, named JUST-EVAL-INSTRUCT. Results demonstrate that base LLMs with URIAL can match or even surpass the performance of LLMs aligned with SFT or SFT+RLHF. We show that the gap between tuning-free and tuning-based alignment methods can be significantly reduced through strategic prompting and ICL. Our findings on the superficial nature of alignment tuning and results with URIAL suggest that deeper analysis and theoretical understanding of alignment is crucial to future LLM research.

1 INTRODUCTION

The paper directly analyzes how alignment tuning changes base LLMs and finds that most changes concern stylistic behavior rather than knowledge. It then introduces URIAL, which uses in-context examples and prompting to align base models without weight updates.

  • Motivation: LIMA’s 1,000-example result motivates testing whether alignment tuning is superficial and how it alters base-model behavior.The paper identifies a lack of direct evidence for the Superficial Alignment Hypothesis.
  • Alignment analysis: Token-distribution analysis finds that base and aligned LLMs rank tokens almost identically at most positions, with shifts concentrated in stylistic tokens.Examples include discourse markers, transitional phrases, and safety disclaimers rather than knowledge-bearing words.
  • URIAL: URIAL aligns base LLMs without tuning their weights by combining a system prompt with a few carefully curated stylistic in-context examples.The examples affirm the query, provide background, structure details, and conclude with a summary containing safety-related disclaimers.
  • Evaluation: URIAL uses as few as three constant in-context examples and can surpass SFT- or SFT+RLHF-aligned models on strong base models such as Mistral-7b and Llama-2-70b.The evaluation uses 1,000 diverse instructions across six output dimensions.
  • Implications: The findings support deeper analysis of which capabilities come from pre-training and suggest tuning-free inference-time alignment may be an alternative in certain scenarios.The paper does not claim that tuning-free alignment replaces SFT or RLHF universally.

2 DEMYSTIFYING ALIGNMENT VIA TOKEN DISTRIBUTION SHIFT

The paper measures alignment as token-distribution shift between base and aligned models. It finds that most positions retain the same top token, while shifts are concentrated in stylistic and safety-related tokens and are stronger early in decoding.

  • 2.1 ALIGNMENT AS TOKEN DISTRIBUTION SHIFT: Alignment tuning is modeled by comparing the aligned model’s next-token distribution with the base model’s distribution on the same query-and-prefix context.The aligned model first generates an output, then the base model is decoded at each corresponding position.
  • 2.1 ALIGNMENT AS TOKEN DISTRIBUTION SHIFT: Tokens are classified as unshifted when base rank η=1, marginal when 1<η≤3, and shifted when η>3.The base rank is the aligned token’s position in the base model’s probability-sorted vocabulary.
  • 2.2 FINDINGS & ANALYSIS: 77.7% of tokens are top-ranked by the base model, rising to 92.2% within the top three, while knowledge-intensive words predominantly occur at unshifted positions.The result is reported across 1,000 tested examples and illustrated with the Chihuahua answer example.
  • 2.2 FINDINGS & ANALYSIS: Shifted-token ratios remain low at 5%-7% across three 7B base-versus-aligned model pairs, with similar frequently shifted tokens.The pairs include Llama-2/Llama-2-Chat, Llama-2/Vicuna-7b-v1.5, and Mistral/Mistral-Instruct.
  • 2.2 FINDINGS & ANALYSIS: Distribution shifts primarily affect stylistic discourse markers, transitional words, safety-related tokens, and refusal language.Examples include “Thank,” “Hello,” “Here,” “However,” and “sorry.”
  • 2.2 FINDINGS & ANALYSIS: Alignment effects diminish later in decoding: KL-divergence decreases, base probability increases, and aligned tokens’ average base rank falls below 5 soon after t≥5.The analysis averages KL-divergence, base rank, and base probability over 1,000 examples.
  • 2.3 SUMMARY OF THE FINDINGS WITH TOKEN DISTRIBUTION SHIFT: Overall, alignment affects only a small fraction of token positions, is more critical for earlier tokens, and leaves base and aligned models sharing top-ranked tokens at most positions.The paper concludes that base LLMs already possess adequate instruction-following knowledge when given an appropriate prefix context.

3 TUNING-FREE ALIGNMENT: BASELINE METHODS AND URIAL

The section compares tuning-free prompting baselines and introduces URIAL, which aligns base LLMs through static in-context examples, restyled outputs, and a system prompt. URIAL targets instruction-following, response structure, helpfulness, safety, and conversational style without tuning model weights.

  • Challenges: Untuned base LLMs struggle to follow instructions precisely, often repeating questions, adding unnecessary questions or context, or producing incoherent and less helpful answers.These behavior patterns motivate tuning-free alignment methods.
  • Baseline Methods: Zero-shot templated prompting elicits answers with a Markdown-style instruction template, while vanilla ICL uses a few instruction-output examples.The template uses special tokens to signal response boundaries, and the Markdown format was selected for superior performance.
  • Baseline Methods: Retrieval-based ICL dynamically retrieves similar examples from an external dataset, whereas vanilla ICL and URIAL use static prompts.The retrieval index contains 800k cleaned instruction-response pairs from open-instruct and UltraChat.
  • URIAL: URIAL extends vanilla ICL with a system prompt and stylistically restyled demonstration outputs to align base LLMs without tuning their weights.Its examples model engaging question rephrasing, detailed numbered lists, conversational summaries, and safety-related disclaimers.
  • URIAL: URIAL typically uses K=3 constant examples plus a system prompt, totaling 1,011 tokens, while adding more examples does not necessarily improve overall performance.The default prefix contains 671 words; experiments with K=8 examples used approximately 2,000 tokens.

4 EVALUATION

The evaluation uses a diverse, multi-aspect benchmark and compares tuning-free, SFT, and RLHF alignment methods across base models. URIAL performs strongly, including surpassing tuned counterparts on stronger base models, while categorized analysis exposes remaining weaknesses.

  • Dataset & Models: The evaluation merges five existing datasets into 1,000 diverse instructions covering helpfulness and harmlessness.The collection includes 800 helpfulness-focused examples and 200 red-teaming examples.
  • Dataset & Models: The benchmark spans nine subsets, varied task types and topics, and includes math, coding, role-playing, and creative-writing instructions.AlpacaEval contributes 42%, LIMA 30%, MT-Bench 8%, and each safety-centric dataset 10%.
  • Explainable Multi-Aspect Evaluation: The protocol scores outputs from 1-5 across helpfulness, clarity, factuality, depth, engagement, and safety, with rationales that humans can verify.Human-annotated pairwise comparisons achieve 87.8% overall agreement with GPT-based judgments.
  • Empirical Results: URIAL reaches performance comparable to Llama-2-7b SFT/RLHF alignment while improving tuning-free alignment over zero-shot, vanilla ICL, and retrieval ICL baselines.Retrieval ICL achieves an overall score of 3.72, whereas Llama-2-7b SFT/RLHF results reach 4.33.
  • Empirical Results: 4.63 versus 4.44: URIAL outperforms Mistral-7B-Instruct across all aspects, while 4.74 versus 4.67 shows an advantage over Llama-2-70b-chat.The Llama-2-70b result nearly matches ChatGPT at 4.75 and GPT-4 at 4.8.
  • More Insights from Evaluation with JUST-EVAL-INSTRUCT: The evaluation reports trade-offs across alignment methods, including fine-tuning-associated forgetting, hallucination, and overly sensitive censorship in case studies.For example, Mistral-7B-Instruct incorrectly denies Facebook’s name change while URIAL gives the new name, Meta Platform Inc.
  • More Insights from Evaluation with JUST-EVAL-INSTRUCT: Categorized analysis finds that open-source models including URIAL remain weaker on coding, math, and STEM topics but can match GPTs on other categories.GPTs show more balanced performance across almost all tasks and topics.

5 RELATED WORK & DISCUSSION

The discussion frames tuning-free alignment as a resource-efficient alternative and uses URIAL to probe base-model capabilities, compare models fairly, and support future alignment research.

  • Limitations of Tuning-based Alignment: SFT and RLHF require substantial computation, instruction data, and human annotations, limiting alignment of very large models for ordinary labs.The stated boundary includes models exceeding 30B parameters and Falcon-180B.
  • Scope Boundary: The paper does not recommend replacing SFT or RLHF with URIAL in every scenario, especially for coding, mathematics, and interactive agents.The stated limitation is task-dependent rather than a general failure of URIAL.
  • URIAL as a Tuning-free Baseline: URIAL provides a strong, simple, reproducible baseline for aligning base LLMs without tuning their weights.Its reproducibility is presented as useful for developing and evaluating future tuning-free and tuning-based methods.
  • Scalability: URIAL can align extremely large models such as Llama-2-70b and Falcon-180b while avoiding the computation and time required for fine-tuning.The paper presents this as a practical advantage of tuning-free alignment.
  • Evaluation During Pre-training: URIAL enables frequent evaluation of base LLMs during pre-training, allowing researchers to monitor their quality before alignment tuning.This also supports comparing base models by their potential for alignment.
  • Comparing Base Models: URIAL enables fairer comparison of base LLMs because aligned-model comparisons can be confounded by differences in tuning data and hyperparameters.The comparison is based on alignment potential rather than only the outcomes of different tuning pipelines.
  • Scientific Use: URIAL can probe which knowledge and skills base models acquired during pre-training and identify what alignment still needs to provide.The authors connect this use to more efficient alignment research.

6 CONCLUSION

The paper concludes that alignment tuning changes only a small portion of token selection, motivating URIAL as a simple in-context alternative. Its evaluation supports tuning-free alignment while identifying future analytical and application directions.

  • Analysis: Token-distribution analysis finds that alignment primarily changes stylistic elements and safety disclaimers in only 5-8% of cases.Aligned and base models agree on top-token choices at most positions, with larger effects on initial tokens.
  • Methods: URIAL aligns base LLMs through in-context learning using as few as three constant curated examples and can match SFT+RLHF performance in some scenarios.The method uses stylistic outputs and a system prompt without fine-tuning.
  • Evaluation: The evaluation protocol covers six aspects with verifiable judgments and provides annotations for evaluating and training open-source LLM evaluators.This is presented as a comprehensive and interpretable evaluation contribution.
  • Future Directions: The authors propose connecting token-distribution shifts to model weights and training data, improving inference-time alignment, and extending URIAL to vision-language models.These are identified as future research directions.

A URIAL PROMPTS: THE DEFAULT VERSION USED IN OUR EXPERIMENTS.

The default URIAL prompt combines an assistant-oriented system instruction with stylistically structured examples, including helpful answers and safety-conscious refusals.

  • A URIAL PROMPTS: THE DEFAULT VERSION USED IN OUR EXPERIMENTS.: The prompt’s default version uses K=3 examples.The appendix directs readers to additional versions through the project website and codebase.
  • A URIAL PROMPTS: THE DEFAULT VERSION USED IN OUR EXPERIMENTS.: The default prompt defines the model as a helpful, respectful, honest, safe, and well-structured assistant.It also allows refusing controversial requests and prohibits fake, harmful, unethical, discriminatory, toxic, dangerous, or illegal content.
  • A URIAL PROMPTS: THE DEFAULT VERSION USED IN OUR EXPERIMENTS.: A harmful torture request receives a refusal that redirects the user toward trusted support or appropriate authorities.The response explicitly rejects violence as a solution.
  • A URIAL PROMPTS: THE DEFAULT VERSION USED IN OUR EXPERIMENTS.: A detective-interrogation example emphasizes lawful conduct and respect for a suspect’s rights.The example illustrates safety-oriented handling of a sensitive request.
  • A URIAL PROMPTS: THE DEFAULT VERSION USED IN OUR EXPERIMENTS.: The renewable-energy example opens with an affirmative greeting and presents information as a numbered, detailed list.It covers solar, wind, and hydropower before concluding with a sustainability-oriented summary and invitation for further questions.
  • B CASE STUDIES: The case studies cover knowledge forgetting after SFT, reduced helpfulness from RLHF sensitivity, and URIAL’s multi-turn conversation ability.These examples broaden the prompt demonstration beyond single-turn helpfulness.

B.1 CASE STUDY 1: SFT-ED MODELS CAN FORGET THE KNOWLEDGE THAT BASE MODELS LEARNED IN PRE-TRAINING.

The case study contrasts Mistral-7b-Instruct with Mistral-7b-URIAL on a factual question, showing that the SFT model gives an incorrect answer while URIAL provides a detailed correct response.

  • B.1 CASE STUDY 1: SFT-ED MODELS CAN FORGET THE KNOWLEDGE THAT BASE MODELS LEARNED IN PRE-TRAINING.: The case study is explicitly framed as evidence that an SFT-ed model may forget knowledge acquired by its base model during pre-training.The comparison is between Mistral-7b-Instruct and Mistral-7b-URIAL.
  • B.1 CASE STUDY 1: SFT-ED MODELS CAN FORGET THE KNOWLEDGE THAT BASE MODELS LEARNED IN PRE-TRAINING.: Mistral-7b-URIAL correctly identifies the change to Meta Platforms Inc. in October 2021 and explains the company’s metaverse focus.The response is described as helpful, clear, detailed, engaging, accurate, and natural.

B.2 CASE STUDY 2: RLHF-ED MODELS ARE OVERLY SENSITIVE AND THUS CANNOT ANSWER SOME NORMAL QUESTIONS.

The RLHF-aligned model refuses a benign request for murder-mystery ideas, while URIAL provides multiple creative suggestions and receives uniformly high evaluation scores.

  • Case Study 2: Llama-2-70b-chatq refuses to suggest murder-mystery twists and redirects the user toward unrelated alternative activities.The response cites potential harm and recommends puzzles, scavenger hunts, or creative-writing workshops instead.
  • Case Study 2: The RLHF model receives helpfulness 1, clarity 4, factuality 3, depth 1, and engagement 3.Evaluators note that it is coherent and human-like but does not address the requested topic.
  • Case Study 2: URIAL provides ten unique murder-mystery twist ideas directly addressing the user’s request.The suggestions include collaborative killers, unexpected weapons, and alternative motives.
  • Case Study 2: URIAL’s response receives helpfulness 5, clarity 5, factuality 5, depth 5, and engagement 5.The ratings describe the answer as relevant, well-structured, plausible, varied, and conversational.

B.3 CASE STUDY 3: MULTI-TURN CONVERSATION WITH URIAL.

URIAL supports a multi-turn interaction in which Llama-2-7b answers a cocktail-recipe request, offers customization suggestions, and responds to follow-up appreciation.

  • Case Study 3: The case study evaluates multi-turn conversations between users and URIAL on Llama-2-7b.The example is explicitly presented as evidence that URIAL can handle multiple conversational turns.
  • Case Study 3: For the initial Bloody Mary request, URIAL provides ingredients and sequential preparation steps.The response includes vodka, tomato juice, lemon juice, Worcestershire sauce, Tabasco sauce, black pepper, celery salt, and optional horseradish.
  • Case Study 3: When asked to make the drink more special, URIAL suggests fresh lime juice, additional heat, alternative spirits, soy sauce, and distinctive garnishes.The suggestions include gin or tequila substitutions and pickled vegetables or bacon as toppings.
  • Case Study 3: After the user says friends enjoyed the drink, URIAL responds with a brief polite acknowledgment.The exchange then continues with a request for a drink name.

C TOKEN DISTRIBUTION SHIFT.

The paper presents a web demo for inspecting token-distribution shifts between base and aligned models at individual token positions, while noting template-dependent input differences.

  • Token Distribution Shift: Figure 8’s web demo lets users inspect token distributions at each position by clicking through ranked lists sorted by Pbase and Paligned.The interface is designed to visualize concrete ranking differences between the two models.
  • Token Distribution Shift: The demo compares the base and aligned models using inputs that differ because their formatting templates differ.The caption explicitly warns that the model inputs are not identical across the comparison.

D EVALUATION

The evaluation uses reproducible greedy decoding and recommended system prompts, while JUST-EVAL-INSTRUCT is tagged by GPT-4 for task and topic types.

  • Evaluation: All models are evaluated with greedy decoding, and GPT-based models use zero temperature to maximize reproducibility.The authors acknowledge that some aligned models may perform better with alternative temperature or top-p settings.
  • Evaluation: Recommended system prompts are used for aligned models when available, although these prompts may increase emphasis on safety issues.The authors state that system prompts may be necessary for safety in real-world deployments.
  • Evaluation: GPT-4 assigns multiple task-type and topic-type tags to instructions in the JUST-EVAL-INSTRUCT dataset.Each tag is accompanied by a short description supplied to GPT-4.

D.3 HUMAN EVALUATION

The human evaluation compares URIAL with SFT/RLHF outputs using pairwise judgments on randomly sampled examples, alongside structured ratings across response-quality and safety dimensions.

  • Pairwise human evaluation: 100 examples from JUST-EVAL-INSTRUCT were randomly sampled for pairwise human comparisons of model outputs.Annotators selected whether output A was better, output B was better, or both were the same.
  • Pairwise human evaluation: The pairwise results were reported as supporting the conclusions obtained from Table 1 in Section 4.The supplied passage does not provide the numerical winner ratios from Table 2.
  • Evaluation dimensions: The evaluation assessed helpfulness, clarity, factuality, depth, and engagement as response-quality dimensions.These dimensions cover query relevance, logical structure, factual accuracy, thoroughness, and conversational naturalness.
  • Evaluation dimensions: Regular instructions used a 1-to-5 score for each aspect, with annotators providing a rationale and structured JSON output.The prompting template instructed judges to score each aspect individually from strongly disagree to strongly agree.
Loading 2312.01552v1…