Nudging: Inference-time Alignment of LLMs via Guided Decoding

1 Introduction

• Training separate aligned versions for every model size within a family incurs substantial computational overhead.The RLHF stage for Tulu 3 405B is given as an example, requiring 11,776 H100 GPU hours.Source: §1 · p. 1
• Prior findings suggest alignment mainly improves helpful, well-formatted responses and affects only a small subset of stylistic tokens.This motivates asking whether training large aligned models is necessary when base and aligned models differ at only a few selected tokens.Source: §1 · p. 1
• NUDGING injects tokens from a small aligned model when the base model’s top-1 token probability falls below a threshold.The method is training-free; prefix caching keeps its added inference overhead minor.Source: §1 · pp. 1–2
• A 7× to 14× smaller aligned model gives zero-shot performance matching, and sometimes exceeding, that of the aligned base model.The paper also reports that NUDGING affects around 10% of output tokens and, with prefix caching, adds about 15% runtime over the base model alone.Source: §1 · p. 2
• The paper frames NUDGING as a modular, cost-efficient direction for token-level collaboration that can reduce alignment costs and increase model flexibility.It reports evaluation across three model families and 13 datasets spanning general knowledge, reasoning, and open-instruction benchmarks.Source: §1 · p. 2

2 Analyzing alignment at the token level

• Alignment-related positions are those where the aligned model’s top token ranks below third in the base model’s distribution.The analysis compares base and aligned model token distributions across 200 examples each from GSM8K, MMLU, and just-eval-instruct.Source: §2 · pp. 2–3

2.1 Where to place nudging tokens?

• When the base model’s top-1 probability is below 0.1, it disagrees with its aligned counterpart over 90% of the time.Across certainty levels, disagreement increases as base-model certainty decreases.Source: §2.1 · p. 3, Figure 2 · p. 3
• A certainty threshold of 0.5 captures over 80% of alignment-related positions while flagging only 11% of all positions.This supports using base-model uncertainty to predict where to place nudging tokens.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2 What to generate as nudging tokens?

• At alignment-related positions, smaller and larger aligned models typically have similar token distributions.On just-eval-instruct, Llama-2-7b-chat’s distribution is similar to Llama-2-70b-chat’s approximately 70% of the time at these positions.Source: §2.2 · p. 3, Table 1 · p. 3

3 Nudging

• NUDGING builds on uncertainty predicting base–aligned disagreement and agreement between aligned models of different sizes at those positions.It is a training-free guided decoding algorithm using a small, off-the-shelf aligned model to generate nudging tokens.Source: §3 · pp. 3–4
• The base model proposes tokens until its top-1 probability falls below threshold γ, then decoding resumes after inserting a nudging-model token.Tokens before the first low-certainty position are accepted, and the base model continues from the updated prefix.Source: §3 · p. 4
• NUDGING uses the nudging model’s first word as the inserted unit and a short lookahead to determine whether generation should stop.Using spaces as boundaries helps standardize collaboration between models with different tokenizers; if the lookahead emits EOS, the full nudging output is appended and generation ends.Source: §3 · p. 4

4.1 Evaluation setup

• Evaluation spans 3 model families and 13 datasets, using zero-shot instructions and greedy decoding; GPT-4o judges answers for most non-math tasks.The datasets cover reasoning, knowledge, open-instruction, and safety tasks.Source: §4.1 · p. 4

4.2 Results on standard benchmarks

• NUDGING performs on par with large aligned models and, for Llama-2 and OLMo, outperforms them on average.Across nearly every dataset and all model families, combining a large base model with a small aligned model gives better performance than either model alone.Source: Table 3 · p. 5, §4.2 · p. 5

OLMo

• Table 3 compares zero-shot instruction-following performance, with smaller aligned models nudging large base models within each family.Its caption reports that nudged large base models perform on par with their aligned counterparts.Source: Table 3 · p. 5

4.2 Results on standard benchmarks

• NUDGING beats other inference-time baselines by 5–10 points across three model families; that comparison averages 11 datasets using 200 samples per dataset.The comparison was conducted on a smaller scale because the baselines were computationally costly.Source: Table 4 · p. 5, §4.2 · pp. 5–6

4.3 Open-ended user instructions and safety

• NUDGING matches aligned models across five open-ended instruction dimensions and surpasses base models; it also covers everyday requests such as writing, role-play, information seeking, and coding.Nudging an unsafe base model with a safe aligned model can greatly enhance its safety.Source: §4.3 · p. 6, Figure 3 · p. 6

4.4 Comparison with In-context Alignment

• On 11 standard benchmarks, NUDGING gains 10.2% for Llama-2 to 24.6% for OLMo over in-context alignment, while remaining competitive on open-ended instructions.A fixed prompt can misclassify benign OLMo queries as unsafe and yield zero accuracy on tasks such as Coin Flip.Source: Table 6 · p. 6, §4.4 · p. 7

4.5 Models from different families

• Using small aligned models from other families boosts Gemma-2-27b by up to 58.6 percentage points; Llama-2-7b-chat nudging largely outperforms Llama-2-70b-chat.Source: §4.5 · p. 7, Table 7 · p. 7

4.6 Scaling up the models

• Scaling the nudging model brings only marginal benefit, whereas using a larger base model brings substantial performance gains.The study scales Gemma-2 models across GSM8K, MMLU, and Arc-challenge.Source: §4.6 · p. 7, Figure 5 · p. 7

• In most cases, about 10% of output tokens come from the nudging model; replacing only 5% of Gemma-2-27b’s tokens recovers 93% of Gemma-2-27b-it’s performance.The reported token ratios are shown across datasets and model settings.Source: §5 · p. 7, Figure 6 · p. 8
• Nudging tokens are mostly formatting markers or discourse cues, while the base model supplies most factual or reasoning content.Examples include “Great” or “1.” for formatting and “In” or “Here” as discourse cues.Source: §5 · p. 8, Table 8 · p. 8
• With prefix caching, nudging Llama-2-70b using Llama-2-7b-chat reaches 0.87× its standalone throughput, adding 15% inference cost.Source: §5 · p. 8
• NUDGING is robust for uncertainty thresholds γ from 0.2 to 0.5; the authors recommend γ = 0.3 or 0.4 as the default.Increasing γ from 0.1 to 0.9 slowly increases the nudging-token ratio.Source: §5 · p. 8

Analysis of alignment

• Prior studies report that alignment affects a small subset of stylistic tokens, can degrade factual or reasoning performance, and primarily adds helpful response behavior to pretrained capabilities.The passage cites Zhou et al. (2024), Chen et al. (2024), Lin et al. (2023), Wang et al. (2023), Ghosh et al. (2024), and Mitchell et al. (2023).Source: §6 · p. 8

Inference-time tuning methods

• Inference-time tuning adapts large models with smaller models, while in-context alignment can shorten usable context and hurt specialized-task performance; NUDGING is described as simpler, faster, and more flexible.The passage also distinguishes NUDGING from Shen et al. (2024), whose token-level collaboration requires task-specific training for each model pair and is not specifically about alignment.Source: §6 · p. 9

Token-level alignment of LLMs

• Token-level alignment work includes token-level rewards for RLHF and decoding-time reward adjustments; NUDGING shares their token-level motivation for adapting model outputs.The passage cites Mudgal et al. (2024), Zeng et al. (2024), and Deng and Raffel (2023).Source: §6 · p. 9

7 Discussion

• NUDGING can harness different models’ strengths while disentangling their capabilities.The discussion presents this modularity as supporting rapid adaptation to new requirements or domains.Source: §7 · p. 9
• A single small aligned model can be reused with newer, more powerful base models, significantly reducing alignment costs.This is especially useful for efficiently aligning very large models or when alignment data is unavailable.Source: §7 · p. 9

8 Conclusion

• NUDGING aligns base models at inference time without additional training, using a lightweight aligned model.Source: §8 · p. 9
• The authors present NUDGING as a lower-cost alternative to traditional alignment, with performance gains across diverse tasks.Source: §8 · p. 9

Limitations

• The approach is motivated by alignment effects at a few token positions, but its reach on complex, multi-part instructions and long contexts remains an open question.The paper offers only a preliminary exploration of those scenarios.Source: Limitations · p. 10
• Its effectiveness for broader alignment concerns, including hallucination and human values, has yet to be established.Source: Limitations · p. 10
• How NUDGING complements post-training for specialized capabilities such as math reasoning or coding remains unresolved.Source: Limitations · p. 10

Ethical Considerations

• The study focuses on instruction following and does not thoroughly evaluate broader alignment issues such as reconciling human values or mitigating hallucinations.Source: Ethical Considerations · p. 10
• The study uses English datasets only; potential discrimination from implicit bias and adversarial use to produce harmful content need further exploration.Source: Ethical Considerations · p. 10

A More implementation details

• The implementation uses API-based and prefix-caching approaches, with experiments run on A6000 GPUs.It sets completion length L to 16 and the maximum token count T to 512.Source: A More implementation details · p. 14, §A.1 · p. 14

A.1 Nudging

• Using the first whole word rather than the first token improves performance and supports collaboration across different tokenizers.The authors hypothesize that words provide more meaningful guidance than subword tokens.Source: §A.1 · p. 14

A.2 Baselines

• The average-ensemble baseline averages top-5 token distributions at each position, while proxy tuning rescales the base distribution using two smaller models.With only five top log probabilities, the models’ top tokens may not intersect, making proxy tuning infeasible for many API providers.Source: §A.2 · pp. 14–15

B Dataset and evaluation details

• The evaluation covers math, commonsense, and symbolic reasoning, academic knowledge, and open-ended instruction and safety tasks.For cost control, the study samples 1,000 test examples per dataset.Source: Reasoning Tasks · p. 15, Knowledge Task · p. 15, Open-Instruction & Safety Task · p. 15

B.2 Evaluation

• Math answers are scored by extracting the final number; other standard benchmarks use GPT-4o comparisons, while instruction and safety tasks follow Lin et al. (2023).The authors manually check that automatic evaluation reflects model performance in general.Source: §B.2 · p. 15

C.1 Inference cost comparison with the baselines

• In wall-clock running-time results, NUDGING is nearly 10x faster than Ensemble and 18x faster than Proxy tuning; both baselines call the base model for every generated token.The authors suggest that API-call count is the most important factor for computational efficiency; calls for later answer tokens reprocess the full prefix.Source: §C.1 · p. 16

C.2 Analysis of the discarded base token ratios in API-based NUDGING

• The discarded base-token ratio is the number of base-model tokens discarded because of the nudging model divided by all tokens generated by the base model.The section relates this efficiency measure to the nudging-token ratio and the completion length per round.Source: §C.2 · p. 16
• With L = 16, the discarded base-token ratio is usually 50%–75%, implying up to 3× extra inference cost; API-call count dominates API-based inference time.The derived upper bound estimates the actual discarded ratio fairly accurately; the authors suggest choosing L more carefully or adapting it to improve efficiency.Source: §C.2 · p. 16, Table 10 · p. 16

E.1 A challenging case

• NUDGING may fail on specialized instructions when neither model represents the needed ability, and inserted tokens can derail a base model’s partial attempt.The authors describe cases where the small model supplies an incorrect response or its token steers the large model away from a partial attempt at satisfying the instruction.Source: §E.1 · pp. 16–17

D Additional results

• When base models are more uncertain, they increasingly disagree with aligned models at the token level.Agreement categories rank the aligned model’s top-1 token as first, second–third, or below third in the base model’s distribution.Source: Figure 8 · p. 17

E.2 Llama-3 results

• On most standard benchmark datasets, NUDGING shows a significant improvement over the large base model.Table 14 reports Llama-3 zero-shot performances on standard benchmarks.Source: §E.2 · p. 17, Table 14 · p. 18
• Nudging Llama-3-70b with Llama-3-8b-instruct underperforms the smaller model alone on many datasets when the instruct model has stronger task-relevant abilities.The authors cite 0-shot CoT MATH performance of 51.9 for Llama-3-8b-instruct versus 41.4 for few-shot Llama-3-70b; they report slight NUDGING improvements over the nudging model on some general-knowledge tasks such as MMLU.Source: §E.2 · p. 17
• On LastLetterConcat, Llama-3-70b-instruct has 2.7% exact-match accuracy, rising to 94% when case differences are ignored.The models use uppercase extracted letters, illustrating a downstream behavior change after alignment or post-training.Source: §E.2 · p. 18

F Visualization of nudging answers

• The appendix presents one nudging answer for each model and dataset, omitting the final [EOS] token.Examples use Llama-2-7b-chat, Gemma-2-2b-it, and OLMo-1b-it for their respective model-family examples.Source: F Visualization of nudging answers · p. 18