Few-Shot Self-Rationalization with Natural Language Prompts

1 Introduction

• Self-rationalization models jointly predict task labels and generate free-text explanations; their explanations are intrinsic to the model.The introduction presents this as a potentially more intuitive way to interact with NLP systems and notes that such models currently require abundant human-authored explanations.Source: §1 · p. 1
• The paper introduces FEB, a standardized collection of four existing English-language datasets with associated metrics, and evaluates natural-language prompts for few-shot self-rationalization.The experiments use T5 and UNIFIEDQA pretrained encoder-decoder transformers.Source: §1 · p. 1
• Versatile question-answering prompts outperform span-infilling prompts by 8.7 accuracy points and prompts modeled on T5’s closest supervised pretraining task by 3.2.These prompting comparisons are part of the paper’s exploration of approaches for few-shot self-rationalization.Source: §1 · p. 2
• Human-rated explanation plausibility and end-task accuracy increase with model size, but GPT-3’s average plausibility remains 25.7 points below human performance.Average plausibility is 50.6 for GPT-3 and 43.4 for UNIFIEDQA-3B.Source: §1 · p. 2

2 FEB Benchmark

• Datasets were selected for textual inputs, short human-authored explanations, a fixed label set, and at least 389 instances; the authors narrowed scope to easier tasks and needed enough data for training and evaluation.The benchmark uses 48 training and 350 evaluation examples; the classification training sets are balanced.Source: §2 · p. 2, Table 1 · p. 2
• FEB includes four datasets spanning natural-language inference, multiple-choice commonsense QA, nonsensical-sentence selection, and offensiveness classification.The benchmark was intended to simplify model comparison and lower barriers to research on few-shot self-rationalization.Source: §2 · p. 2
• COS-E was excluded because the authors considered it too noisy for modeling and evaluation, though they report its results in the appendix.The paper states it does not support future use of COS-E, especially given the introduction of ECQA.Source: §2 · p. 2
• For SBIC, the authors restrict labels to “offensive” and “not offensive” and turn social-bias annotations into short explanatory sentences.Examples include explanations that a post is a personal attack or implies a stereotype about a targeted identity group.Source: §2 · pp. 2–3
• Automatic evaluation uses accuracy for task labels and BERTscore for explanation similarity, assigning zero BERTscore when the predicted label is incorrect.The authors note that BERTscore’s correlation with human judgments of explanation plausibility is weak.Source: §2 · p. 3
• Few-shot evaluation uses 60 random train-dev splits, with 350 examples per dev set and 48 training examples; the authors report means and standard errors across splits.Human evaluation samples six correctly predicted examples per split, balances classification examples by label, and asks three annotators whether explanations justify the answer; workers could not reliably select among ECQA’s likely answers, so that label-check step was skipped for ECQA.Source: §2 · p. 3

3 Prompting for Self-Rationalization

• The approach formats finetuning examples to resemble pretraining data and compares QA, span-filling, and task-matched prompts.The QA prompts use T5 or UNIFIEDQA; the section reports UNIFIEDQA with QASIMPLE as best overall across tasks.Source: §3 · p. 4

3.1 QA Prompts

• With QA prompts, “What is...?” performs best across tasks when input elements are tagged; adding label choices can hurt performance.Tags are especially important for E-SNLI and COMVE, while the “Is...?” prompt is close to random on COMVE regardless of tags.Source: §3.1 · p. 5
• UNIFIEDQA improves on T5 for ECQA and COMVE, while T5 performs better for E-SNLI; gains on SBIC are minor.The authors suggest UNIFIEDQA’s continued QA pretraining may have removed useful NLI information.Source: §3.1 · p. 5

3.2 INFILLING Prompts

• The basic infilling prompt asks T5 to fill a label and explanation after “<extra_id_0> because <extra_id_1>”; natural wording can be prepended.The prompt is flexible enough to add to any task input, and the extra phrase is hypothesized to guide label selection.Source: §3.2 · pp. 5–6, Table 3 · p. 6
• Natural infilling wording benefits ECQA and SBIC, while E-SNLI and COMVE show minor differences favoring the basic prompt.The authors recommend the natural version because it is not detrimental on E-SNLI or COMVE and improves ECQA and SBIC.Source: §3.2 · p. 6, Table 3 · p. 6

3.3 INFILLING vs. ≈T5 vs. QA

• The comparison uses natural INFILLING for T5, task-matched ≈T5 prompts, and QASIMPLE for UNIFIEDQA.For ≈T5, each FEB task is paired with the most similar T5 pretraining task’s format.Source: §3.3 · p. 6, Table 4 · p. 6
• QASIMPLE outperforms the other prompt types on every task except E-SNLI, where ≈T5 performs best.The comparison covers INFILLING, ≈T5, and QASIMPLE prompts.Source: §3.3 · p. 6, Table 4 · p. 6

4.1 Results

• Across datasets, all metrics rise with model size; UNIFIEDQA-3B exceeds GPT-3 on accuracy and BERTscore except SBIC, while GPT-3 explanations are more plausible.Table 5 reports results for T5/UNIFIEDQA model sizes and GPT-3, using task-specific prompts.Source: Table 5 · p. 7, §4.1 · p. 8

4 Improving Self-Rationalization with Increasing Model Size

• The evaluation uses UNIFIEDQA BASE, LARGE, and 3B plus GPT-3 in-context demonstrations, but API cost prevents GPT-3 prompt search.GPT-3 evaluation uses dev sets of 18 examples rather than 350; its demonstrations range from 28 to 45 examples.Source: §4 · pp. 6–7

4.1 Results

• 14.8 points is the average plausibility-score difference between BASE and 3B, with dataset differences ranging from 6.2 to 24.8 points.GPT-3 plausibility ranges from 42.2 to 72.7 across datasets; for SBIC offensive examples it ranges from 42.2 to 52.6.Source: §4.1 · p. 8
• 50.6 for GPT-3 and 43.4 for UNIFIEDQA-3B are average plausibility scores across datasets, versus 76.3 for human-authored explanations.The reported gaps from human explanations are 25.7 points for GPT-3 and 33.0 for UNIFIEDQA-3B.Source: §4.1 · p. 8
• For E-SNLI, entailment plausibility does not scale with model size and its best score is 27.3, versus 66.0/50.9 for the other labels.Human-authored entailment explanations also receive lower plausibility scores than the other two labels.Source: §4.1 · p. 8
• UNIFIEDQA-3B explains SBIC offensive examples best, while GPT-3 explains neutral E-SNLI examples best; SBIC not-offensive explanations are almost perfectly plausible.SBIC’s overall plausibility can obscure larger between-model differences on offensive examples.Source: §4.1 · p. 8

5 Related Work

• Standard free-text explanation methods use human-written explanations, whereas select-then-predict produces highlights without human-authored input highlights (Lei et al., 2016; Liu et al., 2019; Wu and Mooney, 2019; Narang et al., 2020).Source: §5 · p. 8
• Before this work, two cited studies generated free-text explanations with weak supervision: commonsense QA and defeasible inference (Latcinnik and Berant, 2020; Brahman et al., 2021).Source: §5 · p. 8
• This paper introduces FEB to unify few-shot self-rationalization evaluation and presents its first approach and results.Related work also identifies continuous prompts and prefix tuning as possible directions, while noting GPU-memory demands (Qin and Eisner, 2021; Li and Liang, 2021; Ganesh et al., 2021).Source: §5 · p. 9
• Concurrent work studies transfer from a high-resource task, filtering GPT-3 explanation candidates, and generating chain-of-thought explanations before task labels (Wiegreffe et al., 2022; Wei et al., 2022).Source: §5 · p. 9

6 Conclusions

• The paper introduces FEB, a prompting approach established through a comprehensive search of natural language prompts, and results with models from 220M to 175B parameters.Source: §6 · p. 9
• Scaling model size alone increases both generated-explanation plausibility and task accuracy by a very large margin.Source: §6 · p. 9
• GPT-3’s explanation plausibility is 27.7 points below human-authored explanations, so few-shot self-rationalization remains challenging.Source: §6 · p. 9

A.1 Input Formats in Related Work

• The SQUADT5 prompt is identified as T5’s prompt for SQuAD.Source: §A.1 · p. 13
• UNIFIEDQA prompts use a question and context format, with the question-answering prompts serving as their basis.Source: §A.1 · p. 13
• The listed QA formats pair a question with answer choices for multiple-choice QA or with a paragraph for extractive QA.Source: §A.1 · p. 13

• The prompts frame single-text, sentence-pair, and relation classification as questions with multiple-choice answers.Source: UNIFEW · p. 13
• Entity recognition asks for the type of entities marked in the input text.Source: UNIFEW · p. 13

A.2 Prompts Used in §4

• GPT-3 prompts for E-SNLI, ECQA, COMVE, and SBIC pair task-specific inputs with requests to explain or choose an answer.The prompts use hypothesis/premise, question and choices, two candidate sentences, or a post, respectively.Source: §A.2 · p. 13
• The GPT-3 outputs use the form “[answer/label] because [explanation]”.Source: §A.2 · p. 13
• Table 6 maps FEB tasks to similar tasks included in T5 pretraining, whose formats inform prompts compared with span infilling and QA.Source: Table 6 · p. 13
• The authors do not support future use of COS-E, citing reported issues with the dataset and the introduction of ECQA.Source: Table 8 · p. 14, Table 9 · p. 14
• Table 10 compares SQUADT5 question forms and input additions, while Table 11 catalogs COMVE prompt variations across QA, span-filling, and T5-task formats.Source: Table 10 · p. 14, Table 11 · p. 15