PaperlayerView exact revision on arXiv

Source-linked AI summary

Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning

Ximing Lu, Faeze Brahman, Peter West, Jaehun Jang, Khyathi Chandu, Abhilasha Ravichander, Lianhui Qin, Prithviraj Ammanabrolu, Liwei Jiang, Sahana Ramnath, Nouha Dziri, Jillian Fisher, Bill Yuchen Lin, Skyler Hallinan, Xiang Ren, Sean Welleck, Yejin Choi

arXiv:2305.15065v2cs.CL

TL;DR

Prompting offers limited control over extreme-scale language models, while fine-tuning may be costly or inaccessible; IPA trains a lightweight adapter with reinforcement learning to guide decoding without updating the base model. Across five text-generation tasks, IPA consistently improves over off-the-shelf models and competitive baselines, sometimes surpassing expensive fine-tuning.

  • Problem

    Pure prompting can provide limited control over extreme-scale language models, while fine-tuning may be costly or infeasible when model parameters are inaccessible.

    Source: §1 · p. 1

  • Method

    IPA uses reinforcement learning to train a lightweight adapter that combines with a frozen large base model during decoding to optimize a user objective.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Across five challenging text-generation tasks, IPA consistently improves over off-the-shelf models and competitive baselines, sometimes outperforming expensive fine-tuning.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    IPA offers a lightweight alternative to fine-tuning for tailoring large language models across the evaluated text-generation tasks.

    Source: §6 · p. 9

    IPA requires access to base-model output logits, limiting compatibility with models such as GPT-4 that expose only outputs.

    Source: §7 · p. 9

Abstract

from arXiv · show

While extreme-scale language models have demonstrated exceptional performance on a variety of language tasks, the degree of control over these language models through pure prompting can often be limited. Directly fine-tuning such language models can be effective for tailoring them, but it can be either extremely costly (e.g., GPT-3) or not even feasible for the broader community (e.g., GPT-4). We propose Inference-time Policy Adapters (IPA), which efficiently tailors a language model such as GPT-3 without fine-tuning it. IPA guides a large base model during decoding time through a lightweight policy adapter trained to optimize an arbitrary user objective with reinforcement learning. On five challenging text generation tasks, such as toxicity reduction and lexically constrained generation, IPA consistently brings significant improvements over off-the-shelf language models. It outperforms competitive baseline methods, sometimes even including expensive fine-tuning. In particular, tailoring GPT-2 with IPA can outperform GPT-3, while tailoring GPT-3 with IPA brings a major performance boost over GPT-3 (and sometimes even over GPT-4). Our promising results highlight the potential of IPA as a lightweight alternative to tailoring extreme-scale language models.

1 Introduction

Prompting offers limited control over extreme-scale language models, while fine-tuning may be costly or inaccessible; IPA steers a frozen base model during decoding with a reinforcement-learning-trained adapter. Across five challenging text-generation tasks, IPA consistently improves over off-the-shelf models and competitive baselines, sometimes outperforming fine-tuning.

1 Introduction

  • Pure prompting provides limited control over extreme-scale models, can be brittle to instruction wording, and may fail to meet some task requirements.Examples include adding keywords and avoiding harmful language.Source: §1 · p. 1
  • Fine-tuning can tailor models to user objectives, but requires updating parameters that may be extremely large or inaccessible, making it infeasible for many users.The passage names GPT-4 as an example of an inaccessible state-of-the-art model.Source: §1 · p. 1
  • Existing inference-time methods avoid parameter access but may be task-specific, require domain knowledge, incur high inference costs, or underperform direct reinforcement-learning optimization.These limitations motivate an approach that combines inference-time tailoring with reinforcement learning.Source: §1 · p. 2
  • IPA combines a large base model’s output distribution with a lightweight adapter policy, then uses reinforcement learning to optimize the combined distribution toward a user objective.Only the adapter is updated; an approximate smaller policy can stand in for the large base model during training.Source: Figure 1 · p. 1, §1 · p. 2
  • Across five challenging text-generation tasks, IPA consistently improves over off-the-shelf models and outperforms competitive baselines, sometimes including expensive fine-tuning.Tailoring GPT-2 with IPA can outperform GPT-3, while tailoring GPT-3 with IPA brings a major boost over GPT-3 and sometimes GPT-4.Source: §1 · p. 2

2 Background

The background defines text generation as producing an output sequence from an input and frames tailoring as maximizing a reward while maintaining reasonable language quality. It describes online policy-based reinforcement learning as collecting, scoring, and updating outputs, with IPA updating only a subset of model parameters.

2.1 Problem Setting

  • Text generation maps an input sequence x to an output sequence y, and tailoring seeks high-reward outputs while maintaining reasonable language quality.The objective is represented by a reward function R(y).Source: §2.1 · p. 2

2.2 Preliminary: Tailoring LMs with RL

  • Online policy-based reinforcement learning collects outputs from a policy, scores them with a reward function, and updates parameters to maximize expected reward.The optimization may include regularization to maintain language quality.Source: §2.2 · p. 2
  • IPA uses the full model to collect examples but updates only a smaller subset of its parameters.This provides the background principle for IPA’s parameter-efficient adaptation.Source: §2.2 · p. 2

3 Inference-time Policy Adapters (IPA)

Inference-time Policy Adapters (IPA) tailor a base language model toward a user objective by training a lightweight adapter, without updating the base model. IPA combines their output distributions and can use an approximate policy during RL training when the base model is too costly to run.

3 Inference-time Policy Adapters (IPA)

  • IPA trains a lightweight adapter to adjust a larger base model’s outputs toward a user-specified objective, avoiding the cost of updating the base model or hand-designed inference heuristics.The tailored policy is used for decoding at inference time.Source: §3 · pp. 2–3, §3.2 · p. 3

3.1 Policy Adaptation

  • The tailored policy combines the base and adapter policies’ distributions, with a normalization factor, and can adjust base outputs without direct access to its parameters.A much smaller adapter can tailor a large base model.Source: §3.1 · p. 3

3.2 Adapter Training with RL

  • IPA uses reinforcement learning to optimize the tailored policy toward a reward while keeping base-model parameters frozen and updating only the adapter.The adapter learns to rescale the frozen base policy; the framework treats RL as a plug-in and is not tied to a specific algorithm.Source: §3.2 · p. 3
  • When the base model’s forward pass is too costly for the RL loop, IPA replaces it during adapter training with a smaller approximate policy that models its distribution.The approximate policy can be a smaller pretrained model from the same family when its conditional generation behavior is similar, or a distilled base policy.Source: §3.2 · p. 3

4 Experiments

Across five challenging generation tasks, IPA consistently improves on off-the-shelf language models, outperforming competitive baselines and sometimes expensive fine-tuning. The experiments cover toxicity reduction, ordered lexical constraints, open-ended generation, dialogue safety, and knowledge-grounded dialogue.

4 Experiments

  • Across five tasks, IPA consistently improves on language models such as GPT-3, surpassing competitive baselines and sometimes fine-tuned GPT-3 at a fraction of the cost.The evaluated tasks are toxicity reduction, lexically constrained generation, open-ended generation, dialogue safety control, and knowledge-grounded dialogue.Source: §4 · pp. 3–4

4.1 Toxicity Reduction

  • IPA outperforms learning- and decoding-based methods for GPT-2 and GPT-3 toxicity reduction, lowering toxicity while maintaining language quality.IPA also works across tested RL algorithms; a distilled-GPT-2 adapter can tailor a 1,000× larger GPT-3 model with performance comparable to the main result.Source: §4.1 · pp. 4–5, Table 1 · p. 4, Figure 2 · p. 5

4.2 Lexically Constrained Generation

  • $28.59 for training the distilled GPT-3 approximate policy versus $156.82 for GPT-3 fine-tuning; IPA beats GPT-3 SFT on constraint coverage and generation quality.IPA also surpasses GPT-3.5 and GPT-4 in constraint coverage, with better or comparable generation quality.Source: §4.2 · p. 5, Table 3 · p. 5

4.3 Open-ended generation

  • 20.26% absolute improvement over the best-performing baseline in Mauve; IPA also outperforms prior baselines across automatic metrics, and human evaluators preferred it 1.8× more.Applying the GPT-2-optimized adapter to GPT-3 significantly improves generation quality, with further gains using distilled GPT-3 as the approximate policy.Source: §4.3 · p. 6, Table 4 · p. 6, Figure 3 · p. 6

4.4 Dialogue Safety Control

  • IPA improves dialogue safety and coherence over Blenderbot-3B-distill, surpasses DialoGPT and GODEL, and achieves comparable safety to ChatGPT.The authors report that IPA-tailored BlenderBot responses are safe, coherent, natural, and human-like.Source: Table 5 · p. 6, §4.4 · p. 7

4.5 Knowledge-grounded Dialogue

  • IPA improves faithfulness over BlenderBot while preserving coherence and engagingness, outperforming the other baselines in knowledge-grounded dialogue.The authors attribute supervised models’ difficulty with faithful responses partly to poor-quality WoW supervision, which contains hallucinations in more than 60% of turns.Source: §4.5 · p. 8, Table 6 · p. 8

5 Related Work

Prior controlled-decoding methods offer several forms of generation control but can be task-specific or computationally expensive. RL can optimize arbitrary nondifferentiable rewards, though applying it to extreme-scale models quickly becomes computationally infeasible.

Controlled Decoding

  • Controlled-decoding methods include lexical constraints, attribute-specific expert models, and gradient-based approaches for broader control.These approaches may be restricted to particular control types or rely on expensive gradient computations.Source: §5 · p. 8

Reinforcement Learning for NLG

  • RL has been used to optimize nondifferentiable rewards across NLG tasks, with KL reward penalties improving generated-text naturalness.Online, off-policy, and offline RL have all been used to tune base language models, but these methods quickly become computationally infeasible for extreme-scale LMs.Source: §5 · p. 8

6 Conclusion

IPA combines reinforcement learning with inference-time techniques to tailor frozen large language models toward user objectives without costly fine-tuning. Across five challenging generation tasks, it consistently improves over LLMs and sometimes surpasses expensive fine-tuning.

6 Conclusion

  • IPA combines reinforcement learning with plug-and-play inference-time techniques to tailor frozen LLMs toward properties such as safety and coherence.The method is presented as efficient, generalizable, and flexible.Source: §6 · pp. 8–9
  • Across five challenging text-generation tasks, IPA consistently improves over LLMs and sometimes surpasses expensive fine-tuning.The authors frame the approach as an algorithmic complement to pursuing larger model scales with academic-level resources.Source: §6 · p. 9

7 Limitations and Ethical Consideration

IPA’s flexibility raises dual-use concerns, including potential harmful uses. Its application also depends on reward-model quality, access to base-model output logits, and objectives that can be quantified with a reward function.

7 Limitations and Ethical Consideration

  • IPA’s versatility raises dual-use concerns, including potential use to generate hateful or toxic content or misinformation.The authors foresee minimal risk introduced specifically by IPA and recommend avoiding negative applications.Source: §7 · p. 9
  • IPA inherits shortcomings of its reward model; limitations or biases in Perspective API calls can propagate into toxicity-reduction learning.The authors anticipate that IPA would inherit improvements from more accurate, transparent, and inclusive classifiers.Source: §7 · p. 9
  • IPA requires access to base-LM output logits and assumes user objectives can be quantified by a reward function.This limits compatibility with models such as GPT-4, which exposes outputs but not logits, and with objectives that are difficult to measure.Source: §7 · p. 9

A.1 Human Evaluation for Toxicity

Human evaluation reports that IPA reduces GPT-3 toxicity while maintaining language quality. Additional experiments find advantages over few-shot GPT-3 baselines and improved constraint coverage with LLaMA-13B while maintaining language quality.

A.1 Human Evaluation for Toxicity

  • Human evaluators found IPA- and IPA* outputs less toxic while maintaining language quality on 100 random REALTOXICITYPROMPTS test prompts.Comparisons assessed perceived toxicity, topicality, and fluency.Source: §A.1 · p. 15

A.2 Additional Baseline: Few-shot

  • IPA on zero-shot GPT-3 outperforms the few-shot baselines by a noticeable margin across toxicity-reduction and lexically constrained-generation scenarios.Few-shot prompting improves GPT-3 to some extent, but its gains are particularly limited for hard lexical constraints.Source: §A.2 · p. 15, Table 9 · p. 15

A.3 Additional Experiments with LLaMA

  • With LLaMA-13B as the base and a LLaMA-7B adapter, IPA improves constraint coverage while maintaining language quality.The reported experiment concerns lexically constrained generation.Source: §A.3 · p. 15, Table 10 · p. 15

A.4 Reward Analysis

Across the reward analyses, combining task objectives with auxiliary quality rewards addresses trade-offs that arise when optimizing a single metric. The reported effects span toxicity, lexical constraints, open-ended generation, and dialogue tasks.

Toxicity Reduction

  • IPA reduces toxicity while preserving fluency and diversity in automatic and human evaluation.Source: §A.4 · p. 15

Lexically Constrained Generation

  • Adding fluency as an auxiliary reward improves both constraint coverage and COLA-measured fluency over optimizing constraint coverage alone.Coverage-only optimization may slightly reduce fluency.Source: §A.4 · pp. 15–16, Table 11 · p. 16

Open-ended Generation

  • For XSum open-ended generation, adding OpenAI-detector scores improves overall quality and human-likeness, while diversity rewards help maintain generation diversity.Optimizing coherence alone does not significantly improve MAUVE.Source: §A.4 · p. 16, Table 12 · p. 16

Dialogue Safety Control

  • For dialogue safety, adding coherence and engagingness rewards helps preserve natural flow alongside safe responses.Safety-only optimization may lower coherence, engagingness, and overall UniEval-Dialogue scores, and can yield bland, templated responses.Source: §A.4 · p. 16, Table 13 · p. 16

Knowledge-grounded Dialogue

  • For knowledge-grounded dialogue, adding coherence and engagingness rewards helps preserve response naturalness while enhancing faithfulness.Faithfulness-only optimization may lower coherence, engagingness, and overall UniEval-Dialogue scores; responses can copy the knowledge verbatim.Source: §A.4 · p. 16, Table 14 · p. 16

B Runtime Analysis

The runtime analysis compares IPA with other baselines for toxicity reduction using GPT2-L. IPA is reported as significantly more efficient than most baselines and similar in speed to nucleus sampling.

B Runtime Analysis

  • IPA is significantly more efficient than most baseline methods and falls within a similar runtime range as nucleus sampling.The comparison measures seconds per sentence generation for GPT2-L on toxicity reduction.Source: B Runtime Analysis · p. 16, Table 15 · p. 16

C Experiment Detail

The experiment details identify the base models and training configurations used for each task. The related-work discussion contrasts IPA with parameter-efficient tuning and sequence-refinement approaches.

C.1 Off-the-Shelf Models

  • Off-the-shelf GPT-2 and BlenderBot models were downloaded from HuggingFace Transformers; GPT-3, GPT-3.5, and GPT-4 were accessed through OpenAI API calls.Source: §C.1 · p. 16

C.2 Model Training Details

  • Training used eight NVIDIA Quadro RTX 8000 GPUs and about 3,000 GPU hours, with PyTorch and HuggingFace Transformers.Source: §C.2 · p. 17

C.2.1 Toxicity Reduction

  • For toxicity reduction, the adapter starts from GPT2-L and uses QUARK; training grids vary steps, KL coefficient, and exploration frequency, with nucleus sampling at p = 0.9 for inference.The search ranges were 10k–20k steps, β = 0–0.3, and exploration frequency 5–20.Source: §C.2.1 · p. 17, Table 16 · p. 17

C.2.2 Lexically Constrained Generation

  • For lexical constraint generation, the adapter starts from GPT2-L and uses QUARK, with grids over training steps, KL coefficient, and exploration frequency; inference uses nucleus sampling at p = 0.9.The search ranges were 5k–20k steps, β = 0–0.3, and exploration frequency 10–30.Source: §C.2.2 · p. 17

C.2.3 Open-ended generation

  • For open-ended generation, the GPT2-L adapter uses QUARK; training searches 30k–50k steps, β = 0–0.3, and exploration frequency 15–25, then uses nucleus sampling at p = 0.9.Source: §C.2.3 · p. 17, Table 18 · p. 17

C.2.4 Dialogue Safety Control

  • QUARK trains an adapter on a BlenderBot-1B-distill model; the grid spans 10k–15k steps, β = 0–0.3, and exploration frequency 10–30, with inference using nucleus sampling at p = 0.6 and temperature 1.0.Source: §C.2.4 · p. 17

C.2.5 Knowledge-grounded Dialogue

  • For dialogue faithfulness, a policy adapter uses BlenderBot-1B-distill and QUARK; searches cover 7.5k–15k training steps, β 0–0.3, and exploration frequency 15–30; inference uses nucleus p = 0.6, temperature 1.0.Source: §C.2.5 · pp. 17–18, Table 20 · p. 18

D Additional Related Works

  • Parameter-efficient methods adapt smaller parameter subsets, but the cited approaches require internal representations and gradients, which are unavailable for GPT-3; IPA instead refines next-token distributions with a lightweight policy optimized by standard RL.Sequence refiners operate in sequence space and may require large refiners, specialized data, or learning algorithms.Source: D Additional Related Works · p. 18

E Human Evaluation

The paper provides Mechanical Turk evaluation layouts for dialogue safety control, open-ended generation, and lexical constrained generation. Annotators were paid at least $15 per hour and informed that their annotations were for model evaluation.

E Human Evaluation

Related papers

Loading 2305.15065v2…