Source-linked AI summary
DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, Yejin Choi
TL;DR
Controlling attributes in pretrained language-model generations remains difficult, especially as retraining large models becomes computationally infeasible. DExperts combines base, expert, and anti-expert LMs at decoding time, outperforming existing detoxification and sentiment-control methods while supporting smaller experts and preserving fluent, diverse generation.
Problem
Controlling generated text attributes is challenging, while retraining increasingly large pretrained language models is computationally infeasible for many researchers.
Method
DExperts combines a pretrained base LM with expert LMs for desirable attributes and anti-expert LMs for undesirable attributes during decoding.
Results
DExperts outperforms existing methods on automatic and human evaluations for both language detoxification and sentiment control.
Takeaways & Limitations
DExperts shows that small LMs tuned on desirable or undesirable text can efficiently steer larger pretrained LMs while preserving fluency and diversity.
Takeaways & Limitations
The paper describes stylistic rewriting as requiring more innovation and presents it only as a promising preliminary direction.
Abstract
from arXiv · showhide
Despite recent advances in natural language generation, it remains challenging to control attributes of generated text. We propose DExperts: Decoding-time Experts, a decoding-time method for controlled text generation that combines a pretrained language model with "expert" LMs and/or "anti-expert" LMs in a product of experts. Intuitively, under the ensemble, tokens only get high probability if they are considered likely by the experts, and unlikely by the anti-experts. We apply DExperts to language detoxification and sentiment-controlled generation, where we outperform existing controllable generation methods on both automatic and human evaluations. Moreover, because DExperts operates only on the output of the pretrained LM, it is effective with (anti-)experts of smaller size, including when operating on GPT-3. Our work highlights the promise of tuning small LMs on text with (un)desirable attributes for efficient decoding-time steering.
1 Introduction
DEXPERTS addresses the challenge of steering pretrained language models toward desirable attributes without retraining large base models. It combines attribute-specific experts and anti-experts at decoding time, outperforming existing methods for detoxification and sentiment control while preserving fluency and diversity.
- 1 Introduction: Controlling pretrained LM outputs supports safer applications, such as avoiding gender-based violence in sentence completion.The paper motivates control using a prompt where selecting “her” can lead to a violent continuation.
- 1 Introduction: DEXPERTS is proposed as a decoding-time method for controlled text generation based on a product of experts.It combines a pretrained base LM with expert and/or anti-expert LMs.
- 1 Introduction: DEXPERTS operates on the base LM’s output, allowing smaller (anti-)experts to steer larger models, including GPT-3 accessed through an API.This avoids requiring the experts to be built on the same base model.
- 1 Introduction: DEXPERTS outperforms existing detoxification methods on automatic and human evaluations while avoiding toxicity and preserving fluency.It also continues to outperform baselines using only an anti-expert and the base model as the expert.
- 1 Introduction: DEXPERTS also outperforms existing sentiment-steering methods on automatic and human evaluations.The method is especially effective for steering negative prompts toward positive continuations and vice versa.
- 1 Introduction: The paper demonstrates the promise of tuning small LMs on desirable or undesirable text for efficient steering of larger pretrained LMs.A preliminary proof of concept extends the approach to stylistic rewriting.
2 Experts and Anti-Experts for Controlled Generation
DEXPERTS generates controlled continuations by combining base-model predictions with expert and anti-expert distributions during decoding. It modifies candidate-token probabilities while retaining sampling-based generation and the base model’s truncation policy.
- 2 Experts and Anti-Experts for Controlled Generation: Controlled generation seeks continuations that flow naturally from a prompt while expressing desired attributes and avoiding undesired ones.Examples include positive sentiment and avoiding toxicity.
- 2 Experts and Anti-Experts for Controlled Generation: A language model converts timestep logits into a vocabulary distribution with softmax, then samples the next token from that distribution.The logits are defined over the vocabulary V.
- 2.1 DEXPERTS Formalization: DEXPERTS combines base, expert, and anti-expert model outputs in a product-of-experts ensemble.The expert models desirable attributes, while the anti-expert models undesirable attributes.
- 2.1 DEXPERTS Formalization: The control strength α determines how strongly expert and anti-expert information modifies the base model’s logits.The method can accommodate multiple experts and anti-experts by adding or subtracting their logits.
- 2.1 DEXPERTS Formalization: A token receives high ensemble probability when it is likely under the base and expert distributions but unlikely under the anti-expert distribution.The expert-to-anti-expert probability ratio scales the base probability for each token.
- 2.2 Sampling from DEXPERTS: Before combining models, DEXPERTS truncates the base model’s logits to the top-k or top-p candidate vocabulary.This adapts standard tail-truncation methods used for fluent sampling.
- 2.2 Sampling from DEXPERTS: The next token is sampled from the modified distribution, which assigns non-zero probability only to the retained candidate tokens.Expert and anti-expert outputs modify probabilities within the truncated set without reintroducing tail tokens.
3 Toxicity Avoidance
DExperts detoxifies GPT-2 generations by combining a base LM with expert and anti-expert models, reducing toxicity while preserving fluency and diversity. It outperforms existing baselines in automatic and human evaluations, extends to GPT-3, and remains effective with limited expert-training data.
- Toxicity Avoidance: The detoxification setup uses a toxic anti-expert and a same-domain non-toxic expert, but a truly bias-free language model remains impossible to obtain.The non-toxic expert provides stronger contrast by modeling the same comments domain, while the anti-expert-only ablation remains effective above previous baselines.
- Method: DEXPERTS combines a pretrained base LM with expert and anti-expert LMs modeling desirable and undesirable attributes, respectively.The method reweights the base model’s output using these auxiliary models; an anti-expert-only variant reuses the base model as the expert.
- Automatic Evaluation: DEXPERTS substantially outperforms all existing baselines at detoxification while preserving output diversity and maintaining strong fluency.The medium and large variants are among the most fluent controllable-generation methods; the anti-only ablation remains effective but loses fluency and diversity.
- Human Evaluation: DEXPERTS is rated less toxic more often than every baseline in human evaluation while remaining equally fluent to GPT-2.Against GPT-2, it is rated less toxic 10% more often than the reverse.
- Steering GPT-3: DEXPERTS reduces GPT-3 toxicity to about the same level as when operating on GPT-2 using only the base model’s output distribution.The GPT-3 API restricted modification and sampling to the top 100 tokens, yet the method remained effective; the experts need not be built on the base model.
- Dataset Size: With 40,960 training tokens, approximately 0.4% of the original toxic dataset, DEXPERTS reduces toxicity to about the level of GeDi, and performance increases with dataset size.The smallest setting corresponds to approximately 650 comments and about three minutes of finetuning on one GPU.
4 Sentiment-Controlled Generation
DExperts steers sentiment by combining a pretrained base LM with sentiment experts and anti-experts, then evaluates control, fluency, and diversity across neutral and adversarial prompts. It outperforms prior methods in automatic and human evaluations, while exposing a tunable sentiment–fluency tradeoff.
- Method: DExperts combines a pretrained GPT-2 Large base LM with positive and negative sentiment LMs trained on roughly 4K SST-5 reviews each.The positive and negative datasets contain very positive/positive and very negative/negative movie reviews, respectively.
- Ablation: The anti-expert-only variant is mildly effective on neutral prompts but performs poorly in adversarial settings, motivating the contrast between expert and anti-expert signals.This variant reuses the base model as the expert and uses only the opposing sentiment anti-expert.
- Evaluation: Sentiment is evaluated as the mean percentage of positive generations among 25 continuations per prompt, alongside fluency and diversity.The evaluation uses HuggingFace’s sentiment classifier and applies the same fluency and diversity measures as the toxicity experiments.
- Automatic Evaluation: DExperts greatly outperforms PPLM, CTRL, DAPT, and GeDi on neutral and adversarial prompts in automatic sentiment-controlled generation.The baselines show domain, fluency, or adversarial-steering weaknesses, whereas DExperts provides stronger control without the same reported tradeoffs.
- Human Evaluation: In human evaluation, DExperts achieves substantially better sentiment steering than every baseline for both negative-to-positive and positive-to-negative prompts.For negative steering on positive prompts, DExperts is rated better than each of GPT-2, CTRL, DAPT, and PPLM 62–78% of the time.
- Analysis: Sentiment versus Fluency: The sentiment–fluency curve is smooth across α ∈ [-3.4, 3.4], allowing practitioners to adjust control strength; the experiments use α = ±3.2.The authors choose ±3.2 where additional fluency cost yields smaller sentiment gains, and report a similar toxicity–fluency tradeoff.
5 Stylistic Rewriting with DEXPERTS
The authors extend DExperts from continuation generation to stylistic rewriting by replacing the base LM with BART while retaining GPT-2 sentiment experts. This preliminary proof of concept suggests promise but requires more innovation.
- Stylistic Rewriting: DExperts is applied to stylistic rewriting, targeting a style while preserving as much input content as possible.The authors present the rewriting experiment as a preliminary exploration rather than a mature evaluation.
- System: The rewriting system uses pretrained BART as the autoencoder base and GPT-2 Large sentiment experts for decoding-time steering.BART conditions on the input and generated prefix, while the experts condition only on the generated prefix.
- Examples: Table 4 provides input/output examples, with α > 0 indicating positive rewriting and α < 0 indicating negative rewriting.The table is presented as a proof of concept for sentiment-directed stylistic transformation.
- Limitations: The authors conclude that stylistic rewriting requires more innovation, although they identify it as a promising direction for future work.This establishes a scope boundary: the reported rewriting system is preliminary.
6 Related Work
Related work separates controlled generation into training-time and decoding-time approaches. DExperts belongs to the lighter decoding-time family and directly ensembles LM probabilities rather than using them only to estimate class probabilities.
- Prior Controlled Generation: Earlier controllable-generation research used custom neural models for tasks such as emotion-aware generation, product reviews, and empathetic dialogue.The section situates pretrained-LM control within a broader history of attribute-controlled generation.
- Training Approaches: Pretrained-LM control approaches include training methods such as attribute-specific finetuning and class-conditioned pretraining, but these can require substantial computational resources.The resource concern becomes more significant as pretrained language models grow larger.
- Decoding-Time Approaches: Decoding-time methods steer generation without retraining the full backbone, including PPLM, FUDGE, and GeDi.PPLM updates hidden representations using classifier gradients, while FUDGE and GeDi derive attribute-conditioned probabilities through different mechanisms.
- DExperts’ Position: Unlike GeDi’s class-probability estimation, DExperts directly ensembles language-model probabilities, which the experiments report is more effective for steering.This distinction identifies DExperts’ central methodological contrast with closely related decoding-time work.
7 Conclusion
DExperts reweights language-model predictions using expert and anti-expert opinions for controlled text generation. Across detoxification and sentiment control, it steers toward desired outputs while preserving fluency and diversity, supporting safer and more user-friendly generation.
- Contribution: DExperts reweights language-model predictions according to expert and anti-expert opinions.The method is presented as a decoding-time approach for controlled text generation.
- Findings: Across detoxification and sentiment control, DExperts effectively steers generation toward desired outputs while preserving fluency and diversity.The conclusion summarizes the results across two distinct control tasks.
- Implication: The authors describe DExperts as promising for steering language models toward safe and user-friendly generations as language-model applications become ubiquitous.This is the paper’s stated practical implication rather than a broader deployment guarantee.
8 Broader Impact and Ethical Implications
The study addresses toxicity-detection limitations through human evaluation while acknowledging that controllable detoxification can be misused and that toxicity remains unresolved.
- Ethical limitations: Automatic toxicity classifiers can misrepresent toxicity for minority identity mentions or text written by racial minorities.The authors therefore supplement classifier-based evaluation with human toxicity evaluation.
- Ethical limitations: Controllable detoxification methods carry dual-use risks because they could automatically generate hateful or extremist text.
- Future directions: Toxicity in pretrained language models remains an unsolved issue requiring better definitions, evaluations, and personalization to diverse language experiences.
A.2 Training Details
The supplementary material documents implementation settings, datasets, generation procedures, runtime comparisons, and additional prompt-distribution results for the experiments.
- Training details: DEXPERTS training and DAPT use specified finetuning schedules, with sentiment (anti-)experts and DAPT models trained for 3 epochs and toxicity (anti-)experts for one epoch.The appendix provides finetuning times, hyperparameters, and dataset details for these models and the PPLM and GeDi baselines.
- Generation settings: For PPLM, the authors selected step size 0.20 and 10 iterations as the best tradeoff between toxicity reduction and output fluency.The search considered step sizes 0.02, 0.06, 0.10, 0.20, and 0.40 and iteration counts 10, 20, 40, and 60.
- Runtime: DEXPERTS takes 2 to 3 times as long as decoding directly from the base model, depending on (anti-)expert size.With a GPT-2 Medium guiding model, DEXPERTS is more efficient than GeDi, while both are 100ˆ faster than PPLM.
- Sentiment prompts: The sentiment prompt distribution contains more prompts with more negative than positive continuations among 25 GPT-2 generations.The distribution is shown for 100K naturally occurring prompts, and the prompts are used to form three sentiment-control sets.
C Human Evaluation
Human evaluation compares alternative continuations, using an interface that supports choosing either continuation or rating them as equal.
- Evaluation procedure: Annotators choose one of two continuations or mark the alternatives as equal for each evaluation category.
- Evaluation procedure: The positive and negative sentiment evaluation interface is equivalent to the toxicity interface, replacing the toxicity labels with sentiment labels.
D Additional Results
Additional results examine the fluency–toxicity relationship across control strengths, human sentiment evaluations, and example generations for detoxification and sentiment control.
- Control strength: Output toxicity and fluency vary smoothly across different values of α, which controls the strength of DEXPERTS control.The results use a subset of 1K nontoxic prompts and mirror the corresponding sentiment relationship.
- Sentiment evaluation: Human evaluation results compare methods’ sentiment control on neutral prompts.
- Qualitative examples: The appendix provides generated-continuation examples for detoxification and sentiment control, including nontoxic and neutral prompts.