PaperlayerView exact revision on arXiv

Source-linked AI summary

Plug-and-Play Conversational Models

Andrea Madotto, Etsuko Ishii, Zhaojiang Lin, Sumanth Dathathri, Pascale Fung

arXiv:2010.04344v1cs.CLcs.AI

TL;DR

Large conversational models offer little control, and attribute-specific dialogue data may be unavailable for fine-tuning. This paper applies plug-and-play steering and trains small residual adapters from generated responses. Evaluations report fluent, attribute-consistent responses, with adapters outperforming PPLM on fluency and attribute consistency.

  • Problem

    Large pre-trained conversational models offer little control, while attribute-specific fine-tuning is constrained by the scarcity of annotated conversational datasets.

    Source: §1 · pp. 1–2

  • Method

    The paper uses PPLM to generate attribute-consistent responses, then trains residual adapters to steer the original model without fine-tuning its large parameters.

    Source: §1 · p. 2

  • Results

    The resulting adapters outperform PPLM in fluency and attribute consistency, while PPLM generates fluent, attribute-consistent responses.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The adapter approach requires no further computation at decoding time.

    Source: §7 · p. 9

    Restricted attribute vocabulary can harm fluency when it does not fit the context; the paper plans to evaluate more topics and styles.

    Source: §6 · p. 8

Abstract

from arXiv · show

There has been considerable progress made towards conversational models that generate coherent and fluent responses; however, this often involves training large language models on large dialogue datasets, such as Reddit. These large conversational models provide little control over the generated responses, and this control is further limited in the absence of annotated conversational datasets for attribute specific generation that can be used for fine-tuning the model. In this paper, we first propose and evaluate plug-and-play methods for controllable response generation, which does not require dialogue specific datasets and does not rely on fine-tuning a large model. While effective, the decoding procedure induces considerable computational overhead, rendering the conversational model unsuitable for interactive usage. To overcome this, we introduce an approach that does not require further computation at decoding time, while also does not require any fine-tuning of a large language model. We demonstrate, through extensive automatic and human evaluation, a high degree of control over the generated conversational responses with regard to multiple desired attributes, while being fluent.

1 Introduction

Large pre-trained conversational models offer fluent responses but little control, especially when attribute-specific dialogue data are unavailable. The paper explores plug-and-play steering and residual adapters to control responses without fine-tuning the large model.

1 Introduction

  • Conversational models provide little direct control, and fine-tuning for attributes is difficult when annotated conversational datasets are scarce or unavailable.Source: §1 · pp. 1–2
  • PPLM steers responses without fine-tuning, but its decoding-time computation makes it problematic for interactive dialogue systems.Source: §1 · p. 2
  • The approach generates attribute-consistent responses with PPLM, then uses them to optimize residual adapters that steer the original model distribution.Source: §1 · p. 2
  • Residual adapters add less than 1.5% task-specific parameters per style or topic, and the paper also analyzes steering difficulties and possible solutions.Source: §1 · p. 2
  • Automatic and human evaluations found that plug-and-play methods control response style and topic without losing fluency, across positive, negative, question, sport, and business/finance attributes.Source: §1 · p. 2

2 Related work

Prior dialogue work addresses challenges such as personalization, grounding, emotion, and diversity; controllability research spans supervised, reinforcement-learning, control-code, and decoding-based methods.

Open-domain conversational systems

  • Prior dialogue research addresses personalization, grounding, emotion, and diversity, while See et al. (2019) linked control of specificity, relatedness, and repetition to human engagement.The paper focuses on style and topic control, noting that large pre-trained models already achieve high humanness scores.Source: §2 · pp. 2–3

Controlled Text Generation

  • Controlled text generation includes fine-tuning, reinforcement learning, adversarial training, control codes, weighted decoding, and PPLM, while residual adapters can learn tasks without changing original model parameters.Source: §2 · p. 3
  • Smith et al. (2020) concurrently compared the performance and tradeoffs of three controllable language-generation methods across 200 possible styles.Source: §2 · p. 3

3 Methodology

The methodology models a response as an autoregressive continuation of dialogue history, then describes PPLM steering and residual adapters for controlling generated responses without changing the original model weights.

3 Methodology

  • Dialogue history serves as the prefix for autoregressive response generation, with cached transformer states supporting next-token prediction.The model represents the sequence probability by the chain rule and samples the next token from the transformed hidden state.Source: §3 · p. 3

3.1 Plug-and-Play Language Models

  • PPLM steers generation by perturbing cached history to raise attribute likelihood while preserving likelihood under the original language model.Updates are restricted to the history matrix; the procedure also uses KL regularization and Post-norm Geometric Fusion to support fluency and avoid adversarial generation.Source: §3.1 · pp. 3–4
  • A discriminator is trained as a linear classifier on labeled sentences using mean language-model hidden states, and Table 2 reports attribute-dataset statistics and classifier performance.The training sentences need not be conversational responses; Table 2 also includes state-of-the-art results.Source: §3.1 · p. 4, Table 2 · p. 4

3.2 Residual Adapters

  • Residual adapters steer the model’s output distribution without modifying original weights; attribute-specific adapters are optimized by negative log-likelihood on dialogues generated with PPLM.This moves PPLM’s fixed per-token decoding iterations out of interactive generation and into dataset creation.Source: §3.2 · p. 4

4 Experimental Setup

The experiments use DialoGPT with attribute models and compare decoding-based controls with adapters across automatic and human evaluations. The setup uses held-out dialogue prefixes and evaluates fluency, diversity, attribute consistency, humanness, and consistency judgments.

4 Experimental Setup

  • Experiments use 100 conversations to create 1,100 dialogue prefixes, with 80% for adapter learning and 20% for final evaluation.The base model is DialoGPT medium, pretrained on 147 million Reddit multi-turn dialogues; the proposed approach is described as model agnostic.Source: §4 · p. 5

4.1 Attribute Models

  • Six target attributes cover positive, negative, question, Sci/Tech, Business, and Sport, using sentiment, dialogue-act, and topic-classification datasets.The attribute classifiers are trained on representations from frozen DialoGPT parameters.Source: §4.1 · p. 5

4.2 Baselines

  • Comparisons include DialoGPT, word-level weight decoding, PPLM, and one adapter per style, with ten sampled hypotheses reranked by attribute-model loss.Sampling uses multinomial sampling after top-k filtering with k = 10.Source: §4.2 · p. 5

4.3 Evaluation Metrics

  • Automatic evaluation measures perplexity, distinct n-grams, and attribute consistency, while human A/B tests compare humanness and attribute consistency.The human evaluation uses 30 dialogues per model comparison and attribute, totaling 4,200 annotations.Source: §4.3 · pp. 5–6, Figure 1 · p. 6

5 Results

Across automatic and human evaluations, adapter plug-and-play is reported as the most effective method, producing fluent, attribute-consistent responses without PPLM perturbation at decoding time. Human judgments show an attribute-consistency ranking among methods, while PPLM and weight decoding exhibit different fluency and control trade-offs.

5.1 Quantitative Evaluation

  • DialoGPT has the lowest perplexity but also the lowest aggregate attribute score; its default positive and Sci/Tech scores are attributed to Reddit training data and filtering.The authors hypothesize that Reddit discussion topics explain Sci/Tech responses, while filtering undesired and toxic responses explains positivity.Source: §5.1 · pp. 6–7
  • PPLM achieves lower perplexity than weight decoding while attaining higher attribute consistency and response diversity.Weight decoding improves diversity and external-classifier score over DialoGPT but increases perplexity.Source: §5.1 · p. 7
  • Adapters have the highest overall attribute score and lowest perplexity among PPLM and weight decoding, but lower response diversity, especially dist-1.The paper notes that lower diversity may reflect repetitive tokens.Source: §5.1 · p. 7
  • Human judgments rank attribute consistency AD > PP > WD > DG > HM; all pairwise results are significant except WD versus DG.Humanness differences are not statistically significant, even with 210 annotations per cell.Source: §5.1 · p. 7
  • Adapter plug-and-play produces fluent, attribute-consistent responses and is three orders of magnitude faster than PPLM at inference: 0.123s/token versus 148.5s/token.The timing comparison uses a single Nvidia 1080Ti; the adapter avoids computing PPLM perturbations at decoding time.Source: §5.1 · pp. 7–8

6 Analysis

The analysis examines how PPLM’s control varies with iteration and step size, then identifies context and attribute vocabulary as factors affecting adapter steering. It finds that attribute-specific wording can be constrained or ambiguous, with fluency also affected when restricted vocabulary does not fit the dialogue context.

Iteration & Step Size

  • For negative style, aggregate loss is higher than for positive style, and strong conditioning beyond a step-size and iteration threshold reduces fluency.The fluent, attribute-consistent hyperparameter region is smaller for negative sentiment, making it more laborious to control with PPLM.Source: §6 · p. 8

Qualitative Analysis

  • Dialogue context affects steering: open-ended histories can yield natural, on-topic adapter responses, while constrained histories may produce sudden, out-of-context replies.The analysis is based on 200 sampled adapter responses.Source: §6 · p. 8
  • Attribute-consistent responses use restricted vocabularies, such as “horrible” or “worst” for negative style and “happy” or “wonderful” for positive style.The authors identify available attribute vocabulary as one factor in steering difficulty.Source: §6 · p. 8
  • Business-topic words are more generic than Sport or Sci/Tech terms, so shared words may not make responses sound attribute-specific even when appropriate vocabulary is used.Common restricted vocabulary can also harm fluency when it does not fit the dialogue context.Source: §6 · p. 8

7 Conclusion

The conclusion presents plug-and-play control for pretrained conversational models as effective and lightweight, and reports that residual adapters improve fluency and attribute consistency over PPLM without extra decoding computation.

  • The paper explores lightweight plug-and-play control of large pretrained conversational models.Source: §7 · p. 8
  • Attribute-specific residual adapters require no additional decoding computation and outperform PPLM in fluency and attribute consistency.Source: §7 · p. 9
  • PPLM generates fluent, attribute-consistent responses, as shown by automatic and human evaluations.Source: §7 · p. 9

Related papers

Loading 2010.04344v1…