Source-linked AI summary
A Practical Survey on Zero-shot Prompt Design for In-context Learning
Yinheng Li
TL;DR
Prompt design is central to in-context learning, but prompt evaluation is difficult because there is no ground-truth dataset for prompt generation or single best prompt. This survey reviews prompt types and design approaches, including reported findings that carefully engineered zero-shot prompts can outperform few-shot prompts.
Problem
Prompt evaluation is challenging because there is no ground-truth dataset for prompt generation and no single best prompt, while multiple metrics matter.
Source: §4.3 · p. 4, §5 · p. 5Method
Results
Carefully engineered zero-shot prompts can outperform few-shot prompts, and “let’s think step by step” significantly improves mathematical reasoning performance.
Source: §4 · p. 3, §4.1 · p. 4Takeaways & Limitations
Combining manual design, optimization techniques, and rigorous evaluation can support more effective and efficient LLM use across NLP tasks.
Source: §5 · p. 5Prompt performance has no single best choice, and evaluation requires considering multiple metrics.
Source: §5 · p. 5
Abstract
from arXiv · showhide
The remarkable advancements in large language models (LLMs) have brought about significant improvements in Natural Language Processing(NLP) tasks. This paper presents a comprehensive review of in-context learning techniques, focusing on different types of prompts, including discrete, continuous, few-shot, and zero-shot, and their impact on LLM performance. We explore various approaches to prompt design, such as manual design, optimization algorithms, and evaluation methods, to optimize LLM performance across diverse tasks. Our review covers key research studies in prompt engineering, discussing their methodologies and contributions to the field. We also delve into the challenges faced in evaluating prompt performance, given the absence of a single "best" prompt and the importance of considering multiple metrics. In conclusion, the paper highlights the critical role of prompt design in harnessing the full potential of LLMs and provides insights into the combination of manual design, optimization techniques, and rigorous evaluation for more effective and efficient use of LLMs in various NLP tasks.
1 Introduction
In-context learning lets language models use input context to produce responses or predictions, without explicit task-specific training and fine-tuning on labeled data. This survey reviews in-context learning types and design techniques, focusing especially on zero-shot discrete prompts.
1 Introduction
- In-context learning uses input context to generate appropriate responses or predictions, rather than relying on explicit task-specific training and fine-tuning on labeled datasets.The paper identifies zero-shot, one-shot, and few-shot learning as types of in-context learning.Source: §1 · p. 1
- Transformer-based language models have advanced NLU and NLG, while the GPT series shows capabilities across natural-language tasks through in-context learning.The passage names Raffel et al. (2019), Lewis et al. (2019), Brown et al. (2020), and Devlin et al. (2018).Source: §1 · p. 1
- The survey categorizes in-context learning, examines design techniques and advances, and particularly focuses on zero-shot discrete in-context learning.Effective prompt design is described as key to success across in-context learning types.Source: §1 · p. 1
2 Definition
The paper frames prompt design in in-context learning as choosing a prompt that maximizes a scoring function on the test distribution. It distinguishes prompt forms and learning settings, then narrows its review to zero-shot discrete prompts for decoder-only LLMs.
2 Definition
- Prompt design optimization is framed as finding the prompt P that maximizes a model-output score against target labels on the test distribution.The model parameters remain frozen in in-context learning.Source: §2 · p. 1
- Discrete prompts are token lists, whereas continuous prompts are embedding vectors; zero-shot prompts are independent of the input, unlike one- or few-shot prompts.Figure 1 categorizes prompts by form.Source: §2 · pp. 1–2, Figure 1 · p. 2
- The survey focuses on zero-shot in-context learning with discrete prompts in decoder-only LLMs, including the GPT series.Source: §2 · p. 2
3 Relevant Work
The reviewed work spans discrete prompting for BERT, continuous prompt tuning, and few-shot example selection. The survey also notes practical limits: BERT prompt optimization often yields marginal gains, soft prompts require model access, and examples can bias few-shot predictions.
3.1 Prompts for Encoder-only Transformer Models (BERT)
- BERT prompt studies use heuristic, generated, and gradient-searched discrete prompts; prompt optimization generally brings only marginal gains, making downstream fine-tuning more favorable.Examples include Jiang et al. (2020), Gao et al. (2021), and Shin et al. (2020); prompt candidates have also been paraphrased or rewritten, and answers ensembled across prompts.Source: §3.1 · p. 2
3.2.1 Continuous Prompt
- Soft prompts can be optimized as embeddings, but their use requires model-weight access or vector-input capability, limiting feasibility for hosted LLMs such as GPT-3 or PaLM.Prefix-Tuning optimizes prompt embeddings in the transformer; P-Tuning V1 and V2 use a separate LSTM to generate prompts.Source: §3.2.1 · pp. 2–3
3.2.2 Few-Shot Learning
- Few-shot methods select examples similar to test inputs and optimize their order, but examples introduce bias and may hinder inference on out-of-distribution samples.Similarity-based selection has outperformed random selection in the cited studies; similarity may use RoBERTa embedding distances.Source: §3.2.2 · p. 3
4 Zero-Shot Discrete Prompts
Zero-shot discrete prompts are task instructions that can be designed manually or optimized. The reviewed work reports that carefully engineered zero-shot prompts can outperform few-shot prompts in some experiments.
4 Zero-Shot Discrete Prompts
- Carefully engineered zero-shot prompts can outperform few-shot prompts in experiments, though examples do not always help because models may treat them as narrative rather than categorical guidance.Source: §4 · p. 3
- Zero-shot prompts are interpretable, require few examples, simplify design to task instructions, and allow flexible input placement.Source: §4 · p. 3
4.1 Manual Design
- Manual designs include character or dialogue framing and MetaPrompts, alongside principles such as simple output patterns, explicit constraints, and decomposing complex tasks.MetaPrompts express general intentions that take on specific meanings with additional information; subsequent work found some example prompts significantly helpful.Source: §4.1 · p. 4
- Adding “let’s think step by step” to task instructions significantly improves mathematical reasoning, while misleading or irrelevant prompts do not enhance performance.Source: §4.1 · p. 4
4.2 Prompt Optimization
- APE uses LLM-generated candidates and Monte Carlo search, GRIPS edits prompts, FluentPrompt applies a fluency constraint, and RLPROMPT uses reinforcement learning.Prompt optimization typically requires a small evaluation set; FluentPrompt encourages human-readable prompts, whereas RLPROMPT outputs are often incoherent despite claimed performance improvements.Source: §4.2 · p. 4
4.3 Evaluation
- Prompt evaluation lacks a ground-truth prompt set and a uniquely best prompt; likelihood is convenient to calculate but can fail to represent downstream-task performance.Source: §4.3 · pp. 4–5
- Target-task execution accuracy directly measures task performance, while prompt transferability is more useful for comparing design methods than individual prompts.Subset-based estimates can reduce computational cost.Source: §4.3 · p. 5
- Evaluation may also need task-specific risks such as hallucination or toxicity, alongside general language-model metrics such as those in HELM.Prompt design directly impacts model performance.Source: §4.3 · p. 5
5 Conclusion
The conclusion presents prompt design as important across NLP tasks and recommends combining manual design, optimization, and rigorous evaluation. It also notes that prompt evaluation has no single best answer and requires multiple metrics.
5 Conclusion
- In-context learning uses discrete and continuous prompts, as well as few-shot and zero-shot approaches.Source: §5 · p. 5
- Manual design can draw on model behavior, while optimization can search for prompts; evaluation remains challenging because no single prompt is best and multiple metrics matter.Source: §5 · p. 5
- The conclusion recommends combining manual design, optimization techniques, and rigorous evaluation for more effective and efficient LLM use across NLP tasks.Source: §5 · p. 5