PaperlayerView exact revision on arXiv

Source-linked AI summary

Out-of-Distribution Detection and Selective Generation for Conditional Language Models

Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, Peter J. Liu

arXiv:2209.15558v2cs.CL

TL;DR

Conditional language models may degrade unpredictably on out-of-distribution inputs, where autoregressive generation can produce many low-quality sequences. The paper introduces a lightweight embedding-based OOD score and combines it with perplexity for selective generation, demonstrating the approach on summarization and translation while abstaining from lower-quality outputs.

  • Problem

    Conditional language models face OOD inputs where autoregressive errors can compound and arbitrary text generation expands the space of low-quality outputs.

  • Method

    The paper detects OOD examples using input and output embeddings computed during inference, without a separate detection model.

  • Results

    The embedding-based scores significantly outperform baselines on abstractive summarization and translation, and combining OOD scores with perplexity improves selective generation under shifted inputs.

  • Takeaways & Limitations

    OOD scores can support selective generation that produces higher-quality outputs while abstaining from lower-quality ones under input distribution shift.

  • Takeaways & Limitations

    Experiments focus on summarization and translation with encoder-decoder Transformers, although the authors expect broader applicability.

Abstract

from arXiv · show

Machine learning algorithms typically assume independent and identically distributed samples in training and at test time. Much work has shown that high-performing ML classifiers can degrade significantly and provide overly-confident, wrong classification predictions, particularly for out-of-distribution (OOD) inputs. Conditional language models (CLMs) are predominantly trained to classify the next token in an output sequence, and may suffer even worse degradation on OOD inputs as the prediction is done auto-regressively over many steps. Furthermore, the space of potential low-quality outputs is larger as arbitrary text can be generated and it is important to know when to trust the generated output. We present a highly accurate and lightweight OOD detection method for CLMs, and demonstrate its effectiveness on abstractive summarization and translation. We also show how our method can be used under the common and realistic setting of distribution shift for selective generation (analogous to selective prediction for classification) of high-quality outputs, while automatically abstaining from low-quality ones, enabling safer deployment of generative language models.

1 INTRODUCTION

Conditional language models can be especially vulnerable to out-of-distribution inputs because autoregressive errors compound and generation permits many low-quality outputs. The paper proposes lightweight embedding-based detection and selective generation for safer deployment.

  • User inputs in summarization and translation can deviate from training data, violating the IID assumption.
  • Autoregressive errors can propagate, while arbitrary text generation expands the space of possible low-quality outputs.
  • The paper proposes OOD detection methods for conditional language models, evaluated on abstractive summarization and translation.
  • The proposed OOD score uses model input and output representations, requiring negligible additional compute beyond the model itself.
  • Selective generation combines OOD detection with perplexity to generate higher-quality outputs under distribution shift while abstaining from lower-quality ones.

2 OOD DETECTION IN CONDITIONAL LANGUAGE MODELS

Perplexity poorly separates in-domain from OOD examples, motivating a lightweight detector that compares CLM input and output embeddings with training and background distributions.

  • 2.1 PERPLEXITY IS ILL-SUITED FOR OOD DETECTION: Perplexity distributions overlap substantially between in-domain and OOD data for both summarization and translation, making it unsuitable for OOD detection.The overlap is significant for summarization and even greater for translation.
  • 2.2 DETECTING OOD USING CLM’S EMBEDDINGS: The detector averages encoder final-layer states over input tokens and decoder final-layer states over output tokens to form input and output embeddings.These representations are computed during the normal inference or generation process.
  • 2.2 DETECTING OOD USING CLM’S EMBEDDINGS: Mahalanobis distances measure how far test embeddings lie from Gaussian distributions fitted to training input embeddings and ground-truth output embeddings.The input and output sequences are paired, but each representation space receives its own Gaussian model.
  • 2.2 DETECTING OOD USING CLM’S EMBEDDINGS: Relative Mahalanobis Distance extends the classification-based RMD idea to CLMs by contrasting training-domain distances with background-domain distances.The classification formulation cannot be directly applied because CLM outputs are text sequences rather than class labels.
  • 2.2 DETECTING OOD USING CLM’S EMBEDDINGS: Background distributions are fitted using broad datasets, including C4 for summarization and ParaCrawl’s English-French data for translation.The background Gaussian is intended to approximately represent all domains.
  • 2.2 DETECTING OOD USING CLM’S EMBEDDINGS: Decoded outputs can fit the background output Gaussian without supervised document-summary pairs and may accentuate deficiencies on OOD inputs.The resulting contrast is intended to distinguish decoded OOD outputs from in-domain ground-truth labels.
  • 2.2 DETECTING OOD USING CLM’S EMBEDDINGS: RMD scores below zero indicate relatively in-domain examples, whereas positive and larger scores indicate increasing OOD-ness.The method also offers binary logistic-regression scores using input or decoded output embeddings, with higher logits indicating greater OOD likelihood.

3 EXPERIMENTS: OOD DETECTION

Experiments evaluate embedding-based OOD scores for conditional language models on summarization and translation, using AUROC against multiple near- and far-shift datasets. RMD and Binary logits outperform baselines, with RMD better separating near-OOD from far-OOD in summarization.

  • Experiment setup: Experiments use Transformer encoder-decoder models for abstractive summarization and English-French translation, with task-specific in-domain and OOD datasets.Summarization uses PEGASUSLARGE fine-tuned on xsum; translation uses a Transformer trained on WMT15 English-French.
  • Results: RMD and Binary logits achieve high AUROCs above 0.8 and outperform perplexity, NLI, COMET, and Prism, whose scores often fall around 0.5–0.6.The proposed scores perform better overall for OOD detection, while output scores work better for summarization and input scores for translation.
  • Results: Low-quality summarization outputs differ from reference summaries, making output OOD scores more sensitive to their contrast with in-domain references.Examples include repeated text and irrelevant summaries.
  • Results: RMD better distinguishes near-OOD from far-OOD: news datasets lie between in-domain xsum and far-OOD datasets, unlike largely overlapping Binary logits distributions.The near-OOD summarization datasets are cnn dailymail and newsroom; far-OOD datasets include reddit tifu, forumsum, and samsum.
  • Results: All tested methods obtain small AUROC on the translation law dataset, which has the highest unigram overlap with in-domain data at 48.8%.Its surface similarity to the in-domain data may contribute to the low OOD-detection scores.
  • Results: Using C4 sentences as translation background data produces OOD-detection performance very similar to using ParaCrawl.This result suggests limited sensitivity to the choice of background data in that comparison.

4 USING OOD SCORES FOR SELECTIVE GENERATION

The paper evaluates selective generation by combining OOD scores with perplexity to predict output quality and abstain selectively under distribution shift. Across summarization and translation, combined scores generally provide the strongest quality–abstention trade-off.

  • 4 USING OOD SCORES FOR SELECTIVE GENERATION: Selective generation predicts generation quality and abstains when predicted quality is low, extending conservative OOD-based abstention to near-domain inputs.The evaluation uses Quality versus Abstention curves, analogous to accuracy-versus-rejection curves in selective classification.
  • 4.2 PERPLEXITY HAS DIMINISHING CAPABILITY IN PREDICTING QUALITY ON OOD DATA: Perplexity is less reliable for OOD quality estimation, while OOD scores and perplexity provide complementary information.The paper reports that neither score perfectly separates good and bad examples, whereas their combination works much better.
  • 4.4 SELECTIVE GENERATION USING THE COMBINED SCORE: The combined scores achieve the highest quality at almost all abstention rates for both summarization and translation.For translation, OOD scores outperform perplexity when abstention rate α > 0.65, while perplexity is better when α < 0.65.
  • 4.4 SELECTIVE GENERATION USING THE COMBINED SCORE: PRsum survival curves abstain from far-OOD datasets earlier, while near-OOD and in-domain examples remain selected longer.For summarization, reddit tifu and samsum decrease rapidly; cnn dailymail and xsum remain intact until α > 0.3, with xsum surviving longest.

5 RELATED WORK

OOD detection has been studied extensively for classification but less extensively for conditional language models. Existing CLM approaches include embedding- and probability-based methods, with some relying on computationally expensive ensembles.

  • 5 RELATED WORK: OOD detection research began in vision classification and later expanded to text classification tasks including sentiment, inference, intent, and topic prediction.Widely used methods include softmax probability or logit scores and embedding-based distances to the training distribution.
  • 5 RELATED WORK: OOD detection is less studied for conditional language models than for classification models.Prior CLM studies address semantic parsing, speech recognition, and machine translation, but many use ensemble-based methods.
  • 5 RELATED WORK: Ensemble-based CLM OOD methods can require N times the inference cost because they average perplexity over multiple sampled output sequences.This motivates interest in lighter-weight detection approaches.

6 CONCLUSION AND FUTURE WORK

The paper concludes that lightweight OOD scores can support detection and selective generation for conditional language models under input distribution shift. It identifies broader task and architecture applicability as future scope, while suggesting sentence-level OOD attribution as a further direction.

  • 6 CONCLUSION AND FUTURE WORK: Lightweight OOD scores detect out-of-distribution examples for conditional language generation tasks and combine with perplexity to support selective generation.The deployment setting involves generating higher-quality outputs while abstaining from lower-quality ones under input distribution shift.
  • 6 CONCLUSION AND FUTURE WORK: Although experiments focus on summarization and translation, the methods make no assumptions about task modality and may apply to other sequence-output tasks.The paper gives image captioning as an example.
  • 6 CONCLUSION AND FUTURE WORK: The analysis is restricted to encoder-decoder Transformers, although the authors expect the method may also work with decoder-only architectures.The paper names GPT-3, PaLM, and LaMDA as examples of decoder-only systems.
  • 6 CONCLUSION AND FUTURE WORK: Sentence-level attribution of OOD scores is proposed as a possible route toward understanding why particular examples are detected as OOD.The paper points to Section A.13 for one possible attribution method.

A.1 THE OUTPUT QUALITY FOR SUMMARIZATION AND TRANSLATION DATASETS.

The appendix reports output quality for summarization and translation datasets using automatic and human evaluation measures, with dataset-specific sampling and normalization procedures.

  • A.1 THE OUTPUT QUALITY FOR SUMMARIZATION AND TRANSLATION DATASETS.: Summarization quality uses ROUGE-1 over all test samples and human evaluation on 100 samples, with ratings normalized by 5.0 and medianed across three raters.Standard deviations among the three ratings are reported in brackets.
  • A.1 THE OUTPUT QUALITY FOR SUMMARIZATION AND TRANSLATION DATASETS.: Translation quality is reported for each dataset after subsampling all datasets to 1,000 sentence pairs.The table distinguishes summarization and translation evaluation procedures.

A.2 OOD SCORE AND PERPLEXITY ARE COMPLEMENTARY FOR PREDICTING OUTPUT QUALITY.

OOD score and perplexity capture different properties of model outputs, so combining them can improve output-quality prediction.

  • OOD score and perplexity capture different properties of model outputs.
  • Combining OOD score and perplexity can be beneficial for predicting model-output quality.

A.3 AMAZON MECHANICAL TURK ASSESSMENT OF SUMMARY QUALITY

Summary quality was assessed through human ratings collected from Amazon Mechanical Turk workers using a standardized evaluation template.

  • 100 examples from four datasets were evaluated for general summarization quality.The datasets were xsum, cnn dailymail, reddit tifu, and samsum.
  • Three AMT workers rated each example on a 1–5 summarization-quality scale.
  • Workers were US-based Masters with over 95% HIT Approval Rate and at least 1000 approved HITs.They were paid $0.80 per rating.
  • Figure A.2 provides the template used for the summarization human evaluation.

A.4 ALGORITHM FOR RMD OOD SCORES

The RMD OOD-score algorithms fit Gaussian distributions to input and output embeddings from in-domain and background data, then compute scores for test examples.

  • Algorithm 1 fits Gaussian distributions for in-domain and background input embeddings.
  • The method uses a CLM encoder and decoder trained on an in-domain dataset, together with a large background dataset.C4 or ParaCrawl are given as examples of background data.
  • Input embeddings are generated for in-domain training examples before fitting their Gaussian distribution.
  • Output embeddings are generated from model outputs, and separate Gaussian distributions are fit for in-domain and background outputs.
  • Algorithm 2 computes input OOD scores for in-domain and OOD test sets and evaluates them with AUROC.
  • The same inference procedure generates output embeddings, computes output OOD scores, and evaluates their AUROC.

A.5 THE CONNECTION BETWEEN RMD AND BINARY CLASSIFIER

RMD relates Gaussian generative OOD detection to binary classification under shared covariance, but retains greater capacity when class covariances are estimated separately. The section also compares OOD measures, evaluates detection and selective generation, and examines dataset similarity and sentence-level contributions.

  • The connection between RMD and binary classifier: With identical class covariances, the Gaussian generative model can be reduced to a binary classification model.
  • The connection between RMD and binary classifier: RMD estimates each class covariance separately, distinguishing it from a binary classifier and giving it higher model capacity.
  • Comparison with more baseline methods: Logit-based output scores, perplexity, mean(MSP), energy, and ensembled perplexity are generally less competitive than RMD and Binary logits, especially on near-OOD datasets.The ensembled perplexity score costs N times the inference time.
  • Background data: Background data are chosen to be general: C4 for summarization and ParaCrawl for sentence-to-sentence translation.
  • Evaluation: AUROC compares separation of in-domain negatives and OOD positives independently of threshold, with 1.0 indicating perfect separation and 0.5 indistinguishable sets.
  • Dataset similarity: Jaccard n-gram similarity is used to assess whether news datasets are closer to xsum than dialogue datasets.
  • Sentence-level analysis: Sentence-level OOD contribution is estimated by the negative change in article OOD score after removing each sentence.
Loading 2209.15558v2…