Source-linked AI summary
What Makes Good In-Context Examples for GPT-$3$?
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, Weizhu Chen
TL;DR
The paper asks how to reduce GPT-3’s sensitivity to randomly chosen in-context examples. It introduces KATE, which retrieves semantically similar examples for each test sample, and finds consistent gains over random sampling across language understanding and generation tasks, with further gains from task-fine-tuned sentence encoders.
Problem
GPT-3’s empirical results depend heavily on the selected in-context examples, creating a need for better selection strategies than random sampling.
Method
KATE retrieves training examples semantically similar to each test sample and uses them as GPT-3’s in-context examples.
Results
Retrieval-based selection consistently improves GPT-3 over random sampling across several natural language understanding and generation tasks.
Takeaways & Limitations
Task-related fine-tuning of sentence embeddings yields further empirical gains, supporting retrieval as a way to improve GPT-3’s few-shot capabilities.
Takeaways & Limitations
Brute-force search over all possible example combinations is computationally expensive and impractical in many cases.
Abstract
from arXiv · showhide
GPT-$3$ has attracted lots of attention due to its superior performance across a wide range of NLP tasks, especially with its powerful and versatile in-context few-shot learning ability. Despite its success, we found that the empirical results of GPT-$3$ depend heavily on the choice of in-context examples. In this work, we investigate whether there are more effective strategies for judiciously selecting in-context examples (relative to random sampling) that better leverage GPT-$3$'s few-shot capabilities. Inspired by the recent success of leveraging a retrieval module to augment large-scale neural network models, we propose to retrieve examples that are semantically-similar to a test sample to formulate its corresponding prompt. Intuitively, the in-context examples selected with such a strategy may serve as more informative inputs to unleash GPT-$3$'s extensive knowledge. We evaluate the proposed approach on several natural language understanding and generation benchmarks, where the retrieval-based prompt selection approach consistently outperforms the random baseline. Moreover, it is observed that the sentence encoders fine-tuned on task-related datasets yield even more helpful retrieval results. Notably, significant gains are observed on tasks such as table-to-text generation (41.9% on the ToTTo dataset) and open-domain question answering (45.5% on the NQ dataset). We hope our investigation could help understand the behaviors of GPT-$3$ and large-scale pre-trained LMs in general and enhance their few-shot capabilities.
1 Introduction
GPT-3’s few-shot results are highly sensitive to which in-context examples are chosen. This work studies retrieval-based selection of semantically similar examples as a stronger alternative to random sampling.
- Motivation: GPT-3 can generalize to unseen cases from a few demonstrations without task-specific fine-tuning.Its in-context setup concatenates training examples, labels, and a test prompt into one input string.
- Motivation: Different randomly selected contexts can induce substantially different GPT-3 accuracies on the same test set.The paper uses SST-2 sentiment analysis results to illustrate this sensitivity.
- Approach: Nearest examples in embedding space consistently outperform farther examples, motivating retrieval of semantically similar training instances for each test sample.The retrieved examples and test sample are supplied together to GPT-3 for the final prediction.
- Approach: An additional retrieval module selects semantically similar in-context examples and greatly outperforms randomly sampled examples.This strategy is designed to alleviate sensitivity to example choice and better exploit GPT-3’s few-shot capabilities.
- Results: Fine-tuning the retrieval model on task-related datasets produces stronger empirical results with GPT-3.The paper evaluates retrieval-based prompting across natural language understanding and generation tasks, including sentiment analysis, table-to-text generation, and open-domain question answering.
- Results: GPT-3 performance improves as the number of examples available for retrieval increases.This finding identifies retrieval-pool size as an important factor in the method’s effectiveness.
2 Method
The method frames GPT-3 in-context learning as conditional generation and selects semantically similar training examples through sentence-encoder embeddings and k-nearest-neighbor retrieval.
- GPT-3 for In-Context Learning: GPT-3 generates a target y conditioned on source x and a context C containing k labeled examples.The context is formed by concatenating training instances with their corresponding labels.
- The Impact of In-Context Examples: In-context performance is sensitive to example choice, motivating analysis of how example distance from the test sample affects results.The study compares examples that are closer to versus farther from a test sample in embedding space.
- The Impact of In-Context Examples: Nearest neighbors produce much better NQ Exact Match results than farthest neighbors on a subset of 100 test questions.The comparison uses the closest and farthest 10 neighbors, and pre-trained RoBERTa provides effective sentence embeddings.
- kNN-augmented In-Context Example Selection: KATE encodes training and test sources, retrieves each test source’s nearest k training neighbors, and concatenates their sources and targets into GPT-3’s context.The retrieval module uses distances in a sentence encoder’s embedding space and can operate online for test sources.
- Choices of Retrieval Module: The retrieval study compares generally pretrained sentence encoders with encoders fine-tuned on task- or dataset-specific data.The paper motivates fine-tuned encoders as potentially better at assessing similarities relevant to particular tasks.
3 Experimental Setup
The experiments evaluate retrieval-based in-context selection across sentiment analysis, table-to-text generation, and question answering, using task-specific data splits and several baselines.
- Tasks and Evaluation: The evaluation covers sentiment classification, table-to-text generation, and open-domain question answering.GPT-3 uses temperature 0 and generates until a newline token.
- Data Splits: In-context examples are selected from training data; ToTTo and TriviaQA use dev evaluation, while the other datasets use test evaluation.This split reflects leaderboard submission requirements for ToTTo and TriviaQA.
- Sentiment Analysis: Sentiment transfer selects examples from SST-2 and evaluates GPT-3 on IMDB, measuring accuracy with three in-context examples.A RoBERTa-large encoder fine-tuned on SST-2 is also tested, while adding more examples does not improve performance.
- Table-to-Text Generation: ToTTo evaluates descriptions of Wikipedia tables and highlighted cells with BLEU and PARENT, using two in-context examples.Closing angle brackets are removed to fit GPT-3’s 2048-token input limit.
- Question Answering: Open-domain QA evaluates generated answers with Exact Match on NQ, WQ, and TriviaQA, using 64, 64, and 10 retrieved examples respectively.TriviaQA uses 10 examples because 64 would exceed the 2048-token limit.
- Baseline Methods: The random baseline samples training examples, while the kNN baseline predicts from retrieved targets using direct selection or majority voting.KATE and kNN are compared under the same RoBERTa-large embedding space for the kNNroberta baseline.
4 Experimental Results
KATE improves GPT-3’s few-shot performance across sentiment analysis, table-to-text generation, and open-domain question answering by retrieving semantically similar examples. Task-related fine-tuning can improve retrieval, while dissimilar-task fine-tuning can hurt performance.
- Sentiment Analysis: KATE consistently outperforms random example selection on sentiment analysis, with identical results across selections because retrieved examples are fixed.
- Sentiment Analysis: Fine-tuning sentence encoders on similar tasks benefits KATE, whereas fine-tuning on dissimilar tasks can reduce performance.On sentiment analysis, SST-2 fine-tuning yields the best accuracy, while NLI and STS-B fine-tuning can hurt results.
- Table-to-text Generation: KATE improves ToTTo table-to-text results over random selection on both overlap and nonoverlap subsets.The overlap subset shares header names with training data, whereas the nonoverlap subset does not.
- Table-to-text Generation: Retrieved examples provide GPT-3 with detailed table facts, while random examples can produce hallucinated details absent from the source table.The cited examples include points, rebounds, assists, “senior year,” and “University of Texas.”
- Open-domain Question Answering: KATE substantially improves few-shot question-answering accuracy across multiple open-domain QA benchmarks.Fine-tuned transformer encoders improve retrieval over unfine-tuned RoBERTa, and KATE remains better than random selection with only five examples.
- Open-domain Question Answering: On NQ cases, KATE retrieves examples containing correct details or similar question types that help GPT-3 answer questions correctly.
5 Analysis and Ablation Study
The ablation studies examine how the number, training-set source, and ordering of in-context examples affect KATE. Example ordering has data-dependent effects, but its variation is smaller than the gap between KATE and random selection on NQ.
- Number of Examples: On NQ, KATEnli+sts-b is compared with random selection and KATEroberta using 5, 10, 20, 35, and 64 examples.
- Ablation Setup: Figure 3 varies the number of in-context examples and the retrieval training-set size using two representative sentence encoders.
- Order of In-context Examples: The order of retrieved examples is data-dependent: reverse order performs best on NQ, while default order performs slightly better on WQ and TriviaQA.
- Order of In-context Examples: On NQ, order-related variation is small compared with the performance difference between KATE and the random baseline.The authors therefore report that example order does not have a significant impact on KATE’s performance in this setting.
6 Related Work
Prior retrieval-based text-generation systems retrieve exemplars but require task- and data-specific decoders trained from scratch. This work instead studies how semantically similar context can exploit GPT-3 without fine-tuning.
- Retrieval-based Text Generation: Retrieval-based text generation has been applied across translation, sentiment transfer, question answering, dialogue, summarization, data-to-text, and code generation.
- Retrieval-based Text Generation: Earlier retrieve-and-edit frameworks require editor decoders to be trained from scratch, making them task- and data-specific.
- GPT-3 and Retrieval: GPT-3 can function as a universal editor that adapts across tasks without fine-tuning, with more semantically similar context producing better results according to the paper.
- Improve NLP Systems with kNN: Nearest-neighbor methods also augment models using retrieved representations or labels, as illustrated by kNN-LM, kNN-MT, and BERT-kNN.
7 Conclusion
The paper presents KATE as a first investigation of GPT-3’s sensitivity to in-context example selection and retrieves examples by semantic similarity. Across multiple tasks, KATE improves over random sampling, with further gains from task-related sentence-embedding fine-tuning.
- Conclusion: KATE retrieves in-context examples according to their semantic similarity to each test sample.
- Conclusion: KATE significantly improves GPT-3’s performance over random sampling across several natural language understanding and generation tasks.
- Conclusion: Fine-tuning sentence embeddings on task-related datasets produces further empirical gains for retrieval.
- Conclusion: Ablation studies examine robustness to hyperparameters including the number and order of in-context examples.
- Conclusion: The authors position the work as a step toward understanding GPT-3’s behavior and improving its few-shot capabilities.