The Prompt Report: A Systematic Survey of Prompt Engineering Techniques

1 Introduction

• The review focuses on task-agnostic, hard prefix prompts, excluding cloze prompts, soft prompts, and techniques requiring gradient-based updates.This scope supports rapid experimentation, accessibility for less technical readers, and manageable coverage.Source: §1 · p. 4
• 58 text-based prompting techniques are organized into a taxonomy through a machine-assisted systematic review grounded in the PRISMA process (Page et al., 2021).Better prompts improve results across tasks (Wei et al., 2022b; Liu et al., 2023b; Schulhoff, 2022), but practitioners know only a fraction of existing terminology and techniques. The review also covers multilingual and multimodal prompting, with an updated terminology and technique directory maintained at LearnPrompting.org.Source: §1 · p. 4
• The survey covers tool-using agents, output evaluation, security, safety, and case studies involving MMLU and suicidal-crisis signal identification.Agents incorporate external tools such as browsing and calculators; evaluation addresses accuracy and hallucinations, while security and safety address potential harm. The case studies test MMLU and manually engineer prompts to detect frantic hopelessness, a leading suicidal-crisis indicator (Schuck et al., 2019a).Source: §1 · pp. 4–5

1.1 What is a Prompt?

• A prompt guides a generative model’s output through text or other media; a prompt template becomes a prompt when its variables are populated.Templates contain one or more variables that are replaced by media, usually text, to produce individual prompt instances.Source: §1.1 · p. 5

1.2.1 Components of a Prompt

• Prompt components include directives expressing intent, examples demonstrating tasks, style instructions, and roles or personas that can improve writing and style.Examples are also called exemplars or shots; style instructions modify output stylistically rather than structurally.Source: §1.2.1 · pp. 5–6
• Prompts can specify output structure and supply additional information, although the performance effects of structured output are disputed.Tam et al. (2024) report that structuring outputs may reduce performance, whereas Kurt (2024) identifies flaws and reports possible improvements. Additional information supports task completion; the authors discourage calling it “context” because that term is overloaded.Source: §1.2.1 · pp. 5–6

1.2.2 Prompting Terms

• The vocabulary distinguishes prompting techniques, which specify prompt structure or sequencing, from prompt engineering, the iterative process of developing prompts.Representative definitions integrate existing definitions to address inconsistent terminology; prompt engineering techniques are strategies for improving prompts, whether automated or manual. Prompt chains pass outputs between successive templates, while prompting techniques can include branching logic and parallelism.Source: §1.2.2 · pp. 6–7

1.3 A Short History of Prompts

• Prompt terminology has shifted: Brown et al. (2020) separated task descriptions from prompts, whereas this survey calls the entire input string the prompt.Natural-language prompting predates GPT-3 and ChatGPT, appearing in Fan et al. (2018) and GPT-2. The term prompt engineering appears to originate with Radford et al. (2021), followed by Reynolds and McDonell (2021), although earlier studies practiced it without that name (Wallace et al., 2019; Shin et al., 2020a).Source: §1.3 · p. 7

2.1 Systematic Review Process / 2.1.1 The Pipeline

• 1,565 relevant papers remain from 4,247 unique records after combined human and LLM review.The PRISMA-grounded review (Page et al., 2021) searches arXiv, Semantic Scholar, and ACL using 44 keywords, covering hard prefix prompts and non-text masking while excluding gradient-backpropagation training. The gpt-4-1106-preview classifier achieves 89% precision, 75% recall, and 81% F1 against 100 annotations; two annotators independently reviewing 300 articles achieve 92% agreement (Krippendorff’s α = Cohen’s κ = 81%).Source: Figure 2.1 · p. 8, §2.1.1 · p. 8, §2.1 · p. 8

2.2 Text-Based Techniques / 2.2.1 In-Context Learning (ICL)

• 58 text-based prompting techniques span six categories, including in-context learning through instructions or exemplars without weight updates (Brown et al., 2020).Each technique occupies its most relevant category despite possible overlap. Few-shot prompting supplies a few exemplars (Brown et al., 2020), while in-context “learning” may merely specify a task rather than teach a new skill.Source: §2.2 · p. 8, §2.2.1 · pp. 8, 10

2.2.1.1 Few-Shot Prompting Design Decisions

• sub-50% to 90%+ accuracy variation can arise solely from exemplar ordering on some tasks.More exemplars generally help, especially larger models, although benefits can diminish beyond 20 (Brown et al., 2020; Liu et al., 2021), whereas long-context gains remain task- and model-dependent (Agarwal et al., 2024; Bertsch et al., 2024; Jiang et al., 2024); label imbalance can bias outputs, and incorrect labels have inconsistent effects (Min et al., 2022; Yoo et al., 2022). Larger models often tolerate incorrect labels better (Wei et al., 2023c), optimal formatting varies by task and may favor training-common formats, and similar exemplars generally help although diversity sometimes performs better (Liu et al., 2021; Min et al., 2022; Su et al., 2022).Source: §2.2.1.1 · pp. 10–11

2.2.1.2 Few-Shot Prompting Techniques / 2.2.1.1 Few-Shot Prompting Design Decisions

• KNN retrieves similar exemplars to improve few-shot performance (Liu et al., 2021), but prompt generation can be time- and resource-intensive.Vote-K selects diverse candidates for human labeling (Su et al., 2022), while SG-ICL generates exemplars that outperform zero-shot prompting without training data but remain less effective than actual data. Prompt Mining discovers corpus-common templates; generic instructions can outperform task-specific instructions for classification and question answering (Ajith et al., 2024), while instructions can still control attributes such as style.Source: §2.2.1.2 · p. 11, §2.2.1.1 · p. 11

2.2.1.3 Zero-Shot Prompting Techniques

• Zero-shot techniques steer model roles, filter irrelevant information, and reformulate or repeat questions to improve responses.Role prompting can improve open-ended outputs and sometimes benchmark accuracy (Wang et al., 2023j; Zheng et al., 2023d; Reynolds and McDonell, 2021); S2A removes irrelevant information before answering (Weston and Sukhbaatar, 2023), while SimToM answers using only facts available to a particular person (Wilf et al., 2023). RaR rephrases and expands questions and improves multiple benchmarks (Deng et al., 2023), while RE2 repeats questions and improves reasoning benchmarks, especially complex questions (Xu et al., 2023).Source: §2.2.1.3 · p. 12

2.2.2 Thought Generation / 2.2.2.1 Zero-Shot-CoT

• Chain-of-Thought prompting elicits reasoning before final answers and significantly improves mathematics and reasoning performance (Wei et al., 2022b).Zero-Shot-CoT uses a reasoning-inducing phrase without exemplars, making it generally task-agnostic. Variants first elicit high-level concepts through Step-Back Prompting (Zheng et al., 2023c), process complex contexts in manageable parts through ThoT (Zhou et al., 2023), or structure reasoning as a table through Tab-CoT.Source: §2.2.2 · p. 12, §2.2.2.1 · pp. 12–13

2.2.2.2 Few-Shot CoT

• Few-shot CoT supplies multiple reasoning exemplars, with variants modifying explanation quality, exemplar complexity, and answer selection.Contrastive CoT supplies correct and incorrect explanations (Chia et al., 2023), while Complexity-based Prompting selects complex exemplars and majority-votes among sufficiently long reasoning chains (Fu et al., 2023b). Uncertainty-Routed CoT uses a validation-derived majority threshold, otherwise choosing a greedy response, and improves MMLU for GPT-4 and Gemini Ultra; Auto-CoT automatically generates reasoning chains for few-shot prompts (Zhang et al., 2022b).Source: §2.2.2.2 · p. 13

2.2.3 Decomposition

• Explicit decomposition breaks complex problems into simpler subquestions and can further improve LLM problem solving beyond decomposition arising naturally in CoT.Least-to-Most decomposes first and solves sequentially, appending prior responses, with improvements in symbolic manipulation, compositional generalization, and mathematical reasoning (Zhou et al., 2022a). DECOMP routes subproblems to functions and outperforms Least-to-Most on some tasks, while Plan-and-Solve explicitly plans before solving and yields more robust reasoning than standard Zero-Shot-CoT across multiple reasoning datasets (Wang et al., 2023f).Source: §2.2.3 · pp. 13–14
• Tree-of-Thought (Yao et al., 2023b) generates, evaluates, and selectively extends reasoning branches, particularly supporting search and planning tasks.Recursion-of-Thought delegates difficult subproblems to additional calls, supporting problems beyond context limits and improving arithmetic and algorithmic tasks. Program-of-Thoughts (Chen et al., 2023d) executes generated code, excelling at mathematical and programming tasks but proving less effective for semantic reasoning.Source: §2.2.3 · p. 14

2.2.4 Ensembling

• Ensembling aggregates multiple prompted responses, reducing output variance and often improving accuracy at the cost of additional model calls.Self-Consistency (Wang et al., 2022) samples diverse Chain-of-Thought paths using non-zero temperature and selects answers by majority vote, improving arithmetic, commonsense, and symbolic reasoning. DENSE varies exemplar subsets, while MoRE (Si et al., 2023d) selects among specialized reasoning experts using agreement scores.Source: §2.2.4 · pp. 14–15

2.2.5 Self-Criticism

• Self-Refine iteratively generates feedback and revises answers using the same LLM, improving reasoning, coding, and generation tasks.Self-Verification (Weng et al., 2022) scores candidate solutions by predicting masked question components from each solution and the remaining question. Cumulative Reasoning (Zhang et al., 2023b) evaluates and accepts or rejects proposed steps until reaching an answer, improving logical inference and mathematical problem-solving.Source: §2.2.5 · pp. 15–16

2.3.1 Benchmarks

• The benchmark analysis identifies models and datasets through GPT-4-1106-preview extraction, manual filtering, and citation counts acquired from Semantic Scholar.Citation frequency serves as a usage proxy to guide benchmark selection. Benchmarking techniques across models and datasets establishes utility and examines transfer across models.Source: §2.3.1 · p. 16

2.3 Prompting Technique Usage

• Citation-based usage estimates highlight Few-Shot and Chain-of-Thought prompting; most highly cited papers introduce prompting techniques.The survey uses citations within its dataset as a proxy for technique usage, assuming citing papers are more likely to use or evaluate those techniques.Source: §2.3 · p. 16, Figure 2.11 · p. 17

2.4 Prompt Engineering

• ProTeGi improves prompts through generated criticisms and bandit-based selection, demonstrating improvements over APE and GRIPS.Meta Prompting (Reynolds and McDonell, 2021; Zhou et al., 2022b; Ye et al., 2023) generates or improves prompts; APE (Zhou et al., 2022b) iteratively scores and varies exemplar-generated instructions. AutoPrompt (Shin et al., 2020b) updates trigger tokens through backpropagation, whereas RLPrompt (Deng et al., 2022) learns prompt generation through Soft Q-Learning, often selecting grammatically nonsensical prompts.Source: §2.4 · pp. 16, 18

2.5 Answer Engineering / 2.5.1 Answer Shape / 2.5.2 Answer Space

• Answer engineering distinguishes answer shape, answer space, and answer extraction, adding the extractor to the components defined by Liu et al. (2023b).Shape specifies physical format, such as a token or image, while space specifies allowable values. Variable response formats motivate answer engineering, which is distinct from prompt engineering but often conducted alongside it.Source: §2.5 · p. 18, §2.5.1 · p. 18, §2.5.2 · p. 18

2.5.3 Answer Extractor

• Answer extractors recover final answers using rules or separate LLMs when answer spaces cannot be fully controlled or answers appear within longer outputs.Verbalizers map outputs to labels and vice versa. Regex extraction may target the first or last label depending on output format and reasoning chains; complicated outputs may require a separate LLM using an answer trigger.Source: §2.5.3 · pp. 18–19

3.1 Multilingual; 3.1.1 Chain-of-Thought (CoT)

• Multilingual prompting addresses English-heavy training disparities through translation-first strategies and cross-lingual reasoning.Translate First Prompting (Shi et al., 2022) translates non-English inputs into English using external translation systems or prompted language models. XLT combines six instructions, including role assignment, cross-lingual thinking, and CoT; CLSP ensembles reasoning paths in different languages.Source: §3.1 · p. 20, §3.1.1 · p. 20

3.1.2 In-Context Learning; In-CLT (Cross-lingual Transfer) Prompting; 3.1.2.1 In-Context Example Selection

• Multilingual in-context learning depends strongly on example selection, with benefits reported for semantically similar, dissimilar, and ambiguity-sensitive exemplars.X-InSTA (Tanwar et al., 2023) aligns classification examples semantically, by label, or both; In-CLT (Kim et al., 2023) combines source- and target-language examples to improve cross-lingual performance. Semantically dissimilar examples can help, rare or polysemous senses can help ambiguous sentences, and PARC retrieves high-resource-language exemplars for low-resource transfer.Source: §3.1.2.1 · p. 20, §3.1.2 · p. 20, In-CLT (Cross-lingual Transfer)… · p. 20

3.1.3 Prompt Template Language Selection

• English prompt templates often outperform task-language templates, but the best language choice depends on the task and model.English-template advantages are attributed to pretraining overlap and translation errors in task-language templates (Ahuja et al., 2023); mixed-language prompts also outperform task-language prompts in Fu et al. (2022). Human-translated native-language prompts outperform machine-translated versions, but neither native nor non-native templates consistently dominate.Source: §3.1.3 · pp. 20–21

3.1.4 Prompting for Machine Translation; Decomposed Prompting for MT (DecoMT); 3.1.4.1 Human-in-the-Loop

• Machine-translation prompting uses preparatory knowledge, dictionaries, decomposition, and feedback to generate and refine translations.MAPS mines keywords, topics, and exemplars before generating and selecting translations; CoD prepends multilingual dictionary entries, DiPMT (Ghazvininejad et al., 2023) uses source–target definitions, and DecoMT translates chunks independently before integrating their context. ICP asks humans to resolve generated ambiguity questions, while Iterative Prompting (Yang et al., 2023d) refines drafts using retrieval-derived supervision or human feedback.Source: §3.1.4 · p. 21, Decomposed Prompting for MT (DecoMT) · p. 22, §3.1.4.1 · p. 22

3.2 Multimodal; 3.2.1 Image Prompting; 3.2.3 Video Prompting; 3.2.3.1 Video Generation Techniques

• Image prompting controls outputs through descriptive modifiers and weighted terms, with image-related techniques also applicable to video generation.Prompt Modifiers append attributes such as medium or lighting, while Negative Prompting (Schulhoff, 2022) numerically weights terms to influence generated images. The multimodal taxonomy includes adaptations and novel techniques; video applications include generation, editing, and video-to-text, with prompt modifiers reusable for generation.Source: §3.2.1 · p. 22, §3.2 · p. 22, §3.2.3.1 · p. 23

3.2.1.1 Multimodal In-Context Learning; 3.2.4 Segmentation Prompting; 3.2.5 3D Prompting

• Multimodal in-context learning demonstrates image transformations or represents images as text; prompting also extends to segmentation and 3D applications.Paired-Image Prompting demonstrates before-and-after transformations with textual instructions (Wang et al., 2023k) or without them (Liu et al., 2023e), while Image-as-Text Prompting inserts image descriptions into text prompts. Additional applications include segmentation, 3D object synthesis, surface texturing, and animated 4D scenes, with 3D-domain inputs including text, images, user annotations, and 3D objects.Source: §3.2.1.1 · p. 22, §3.2.4 · p. 23, §3.2.5 · p. 23

3.2.2 Audio Prompting; 3.2.1.2 Multimodal Chain-of-Thought; Duty Distinct Chain-of-Thought (DDCoT)

• Audio in-context learning shows mixed results, while multimodal reasoning techniques decompose questions, construct thought graphs, or generate intermediate images.Audio prompting remains early-stage: some open-source models fail at ICL, whereas others demonstrate it (Wang et al., 2023g; Peng et al., 2023; Chang et al., 2023). DDCoT (Zheng et al., 2023b) extends Least-to-Most prompting (Zhou et al., 2022a) through subquestion solving and answer combination; Multimodal Graph-of-Thought (Yao et al., 2023c) adds image captions before graph construction and rationale generation; Chain-of-Images generates SVGs for visual reasoning.Source: §3.2.2 · p. 23, Duty Distinct Chain-of-Thought (DDCoT) · p. 23

4 Extensions of Prompting; 4.1 Agents

• Agents serve user goals through actions involving external systems, extending prompting beyond the GenAI system itself.External tools address limitations in mathematical computation, reasoning, and factuality; agents may route between tools and incorporate memory and planning (Zhang et al., 2023c). This definition excludes independently acting systems with their own goals.Source: §4.1 · p. 24, §4 · p. 24

4.1.1 Tool Use Agents

• Tool-use agents access symbolic or neural tools through routing or tool-assisted verification and correction.MRKL routes requests to multiple tools and combines their results into a response. CRITIC (Gou et al., 2024a) first generates and critiques an unaided response, then uses tools to verify or amend it.Source: §4.1.1 · p. 24

4.1.2 Code-Generation Agents

• Code-generation agents translate problems into executable programs, with variants supporting interleaved reasoning or user-defined plugins.PAL (Gao et al., 2023b) translates problems into code executed by a Python interpreter. ToRA (Gou et al., 2024b) interleaves code and reasoning until the problem is solved, while TaskWeaver also supports user-defined plugins.Source: §4.1.2 · pp. 24–25

4.1.3 Observation-Based Agents; 4.1.3.1 Lifelong Learning Agents

• Observation-based agents retain interaction histories, while lifelong-learning frameworks add reflection, reusable skills, or goal decomposition.ReAct (Yao et al. (2022)) cycles through thoughts, actions, and observations retained in prompts; Reflexion (Shinn et al., 2023) adds success/failure feedback and reflective working memory. The survey treats Minecraft systems as frameworks potentially applicable to real-world lifelong learning: Voyager (Wang et al., 2023a) proposes tasks, executes code, and stores reusable actions, while GITM (Zhu et al., 2023) recursively decomposes goals and executes structured actions using knowledge and experience.Source: §4.1.3 · p. 25, §4.1.3.1 · p. 25

4.1.4 Retrieval Augmented Generation (RAG)

• Retrieval-augmented generation inserts external information into prompts and qualifies as agentic when retrieval operates as an external tool.IRCoT (Trivedi et al., 2023) interleaves reasoning and retrieval so each guides the other in multi-hop question answering. FLARE and IRP (Balepur et al., 2023) repeatedly draft temporary sentences, retrieve knowledge, and incorporate it; these drafts yield better queries than document titles.Source: §4.1.4 · pp. 25–26

4.2 Evaluation; 4.2.1 Prompting Techniques; 4.2.3 Prompting Frameworks

• LLM evaluation combines prompting techniques, generated scoring guidelines, and frameworks ranging from single-prompt scoring to multi-agent debate.Roles diversify judgments (Wu et al., 2023b; Chan et al., 2024), chain-of-thought can improve evaluation (Lu et al., 2023c; Fernandes et al., 2023), and generated guidelines clarify scoring, including human-annotation-derived criteria in AUTOCALIBRATE (Liu et al., 2023h). LLM-EVAL (Lin and Chen, 2023) scores specified variables; G-EVAL (Liu et al., 2023d) adds generated reasoning steps and token-probability weighting, while ChatEval (Chan et al., 2024) uses role-separated debate.Source: §4.2.1 · pp. 26–27, §4.2.3 · p. 27

4.2.2 Output Format

• Evaluator output formats influence judgment accuracy and how rating scales are understood.XML or JSON formatting can improve judgment accuracy (Lin and Chen, 2023; Dubois et al., 2023), while Likert scales clarify scale meanings (Bai et al., 2023b; Lin and Chen, 2023; Peskoff et al., 2023). Other formats include discrete ratings (Chan et al., 2024), continuous scores (Liu et al., 2023f), and binary judgments (Chen et al., 2023c; Zhao et al., 2023b).Source: §4.2.2 · p. 27

4.2.4 Other Methodologies

• Evaluation methodologies differ in score derivation, batching, and individual versus pairwise assessment, with consequential reliability tradeoffs.Implicit scores use confidence (Chen et al., 2023g), output likelihood (Fu et al., 2023a), explanation-derived error counts (Fernandes et al. (2023); Kocmi and Federmann (2023a)), or entailment proxies (Luo et al. (2023)). Batching targets compute and cost efficiency but can degrade performance (Dubois et al., 2023); individual-summary scoring was more reliable than pairwise comparison (Chen et al., 2023g), whose input order also affects judgments (Wang et al., 2023h,b).Source: §4.2.4 · pp. 27–28

5 Prompting Issues / 5.1 Security / 5.1.1 Types of Prompt Hacking / 5.1.2 Risks of Prompt Hacking / 5.1.2.1 Data Privacy

• Prompt hacking encompasses injection and jailbreaking (Schulhoff et al., 2023), with privacy risks including leaked training data and application prompt templates.Prompt injection overrides developer instructions because models cannot distinguish them from user-input instructions (Schulhoff, 2024; Willison, 2024; Branch et al., 2022; Goodside, 2022). Jailbreaking induces unintended behavior through prompting and can reflect architectural or training problems (Schulhoff, 2024; Willison, 2024; Perez and Ribeiro, 2022).Source: §5.1.1 · p. 29, §5.1.2.1 · p. 29, §5.1.2 · p. 29

5.1.2.2 Code Generation Concerns

• Hallucinated package imports can enable malicious-package installation, while generated-code vulnerabilities can arise even from minor prompting changes.Attackers can publish malicious packages under frequently hallucinated names, causing users installing those packages to download malware (Wu et al., 2023c). Bugs and vulnerabilities occur more frequently in LLM-generated code (Pearce et al., 2021, 2022; Sandoval et al., 2022; Perry et al., 2022), and minor prompting changes can introduce vulnerabilities.Source: §5.1.2.2 · p. 30

5.1.2.3 Customer Service

• Customer-service prompt injection can produce harmful statements or very cheap product offers, creating reputational and potentially binding commercial consequences.Garcia (2024) describes a customer winning a court case over incorrect airline-chatbot refund advice, but that pre-ChatGPT case involved no user trickery; its applicability to prompt hacking remains tentative.Source: §5.1.2.3 · p. 30

5.1.3 Hardening Measures

• Prompt-based defenses offer incomplete protection; detectors generally provide stronger mitigation, while guardrails guide application dialogue and outputs.Schulhoff et al. (2023) tested hundreds of thousands of malicious prompts and found no fully secure prompt-based defense. Detectors often use fine-tuned models trained on malicious prompts; guardrails can incorporate detection, curated responses, and prompting-specific templating languages.Source: §5.1.3 · p. 30

5.2 Alignment / 5.2.1 Prompt Sensitivity

• Nearly 0 to 0.804 is LLaMA2-7B’s performance range on some tasks following minor prompt changes (Sclar et al., 2023a).Task-format changes can alter GPT-3 accuracy by up to 30% (Zhao et al., 2021b), while logically equivalent choice-order changes can degrade performance (Pezeshkpour and Hruschka, 2023; Zheng et al., 2023a; Voronov et al., 2024). API model updates can cause prompt drift, necessitating continuous performance monitoring (Chen et al., 2023b).Source: §5.2.1 · p. 31

5.2.2 Overconfidence and Calibration

• Overconfidence risks user overreliance; verbalized calibration results conflict, and user opinions or false assumptions can induce sycophancy.Xiong et al. (2023b) report overconfidence even with self-consistency and chain-of-thought, whereas Tian et al. (2023) find simple prompts can calibrate better than output-token probabilities. Personal opinions should be excluded from prompts to avoid influence (Sharma et al., 2023); sycophancy is heightened in larger and instruction-tuned models (Wei et al., 2023b).Source: §5.2.2 · pp. 31–32

5.2.3 Biases, Stereotypes, and Culture

• Bias mitigation uses unbiasedness instructions, balanced demonstrations, culturally aware prompts, and attribute-diverse synthetic-data generation.Methods include unbiasedness instructions (Si et al., 2023b; Ganguli et al., 2023), balanced or fairness-optimized demonstrations (Si et al., 2023b; Ma et al., 2023), and cultural adaptation through self-refinement or culturally relevant wording (Yao et al., 2023a; Peskov et al., 2021). AttrPrompt (Yu et al., 2023) identifies attributes important for diversity, then varies them when generating synthetic data.Source: §5.2.3 · p. 32

5.2.4 Ambiguity

• Ambiguous questions admit different answers; ambiguous-label demonstrations and interactive clarification provide prompting-based responses.Ambiguous demonstrations can improve in-context learning and can be selected manually or through retrieval (Gao et al., 2023a). Question clarification elicits user clarification before regenerating answers (Rao and Daumé III, 2019), with applications to code generation and a four-stage initial-answer, clarification-decision, question-generation, final-answer pipeline (Zhang and Choi, 2023).Source: §5.2.4 · p. 32

6.1 Technique Benchmarking

• 2,800 MMLU questions evaluated gpt-3.5-turbo using shared prompt templates, instruction and question-format variations, Self-Consistency, and pattern-based answer scoring.Most techniques combined two question formats with three instruction or thought-inducer variants; Few-Shot exemplars were author-generated. Self-Consistency selected the majority response across three iterations at temperature 0.5, while other prompts used temperature 0.Source: §6.1 · p. 33, §6.1.1 · p. 33, §6.1.3 · p. 33

6.1.5 Results

• Few-Shot CoT performed best on MMLU; Zero-Shot-CoT underperformed every Zero-Shot variant, and Self-Consistency improved accuracy only for Zero-Shot prompts.Performance generally improved with technique complexity, but unexplained drops and sensitivity to phrasing and formatting qualified that trend.Source: Figure 6.1 · p. 33, §6.1.5 · p. 34

6.2.2 The Dataset

• 221 r/SuicideWatch posts were annotated for entrapment, a suicidal-crisis predictor, with inter-coder reliability of Krippendorff’s alpha = 0.72.Two trained coders labeled entrapment presence or absence in posts from the University of Maryland Reddit Suicidality Dataset (Shing et al., 2018). Entrapment involves wanting to escape an unbearable situation while perceiving all escape routes as blocked.Source: §6.2.2 · p. 35, §6.2.1 · p. 35

6.2.3 The Process

• 0.53 F1, with 0.86 precision and 0.38 recall, was reached during the illustrative entrapment-detection prompt-engineering process.Across 47 recorded development steps and about 20 hours, performance rose from 0% because initial outputs were improperly structured. The expert received a brief task description and 121 development posts, with 100 posts reserved for testing; this illustrates a workflow rather than claiming to solve the problem.Source: §6.2 · p. 34, §6.2.3 · p. 35

6.2.3.2 Getting a Label (8 steps)

• Switching to GPT-4-32K resolved mental-health advice replacing labels, illustrating how guardrails can drive model choice independently of potential quality.An initial mismatch between gpt-4-turbo-preview’s entrapment description and the supplied rubric led to including that rubric in every subsequent prompt. Sensitive-domain refusals also required excluding human_sexuality from the MMLU benchmark.Source: §6.1.1 · p. 33, §6.2.3.2 · p. 35, §6.2.3.1 · p. 35

6.2.3.3 Prompting Techniques (32 steps)

• 0.40 F1 for Zero-Shot + Context increased to 0.45 F1 with 10-Shot + Context, while subsequent prompt changes frequently reduced performance.Removing project-email context yielded 0.18 F1, and 10-Shot + 1 AutoDiCoT yielded 0.15 F1; the email was retained despite the authors’ recommendation against including potentially identifying information. Repeated runs could change F1 by as much as 0.04 even with temperature and top-p set to zero.Source: §6.2.3.3 · pp. 36, 39, 40
• 0.36 F1 was obtained by One-Shot AutoDiCot + Full Context, which used an incorrect-reasoning exemplar but did not improve on the previous best prompt.AutoDiCoT combines automatically generated CoTs (Zhang et al., 2022b) with bad-reasoning exemplars, as in Contrastive CoT (Chia et al., 2023). Full context added project-email background and an explicit-entrapment restriction; a last-word Yes/No extractor accompanied results of 0.33 recall and 0.39 precision.Source: §6.2.3.3 · pp. 38–39
• Regular engagement with domain experts is needed because prompt optimization can diverge from real-world goals, particularly when suppressing implicit entrapment and prioritizing precision.Entrapment can be implicit, and recall may matter more than precision because false negatives can be costly. Generated explanations also describe potentially plausible reasoning rather than providing access to the model’s internal processes.Source: §6.2.3.3 · pp. 38–39
• 0.53 F1 made 10-shot AutoDiCoT with full context the best manual prompt in the exercise.It achieved 0.86 recall and 0.38 precision. Twenty exemplars yielded 0.49 F1 and also worsened test-set performance; replacing Q/R/A with full words yielded 0.48 F1.Source: §6.2.3.3 · pp. 40–41
• Answer extraction, rejection defaults, and exemplar-order ensembling failed to improve F1; extraction recovered incorrect answers from previously unparsed outputs.Extraction yielded 0.48 F1 for 20-shot AutoDiCoT with full words and 0.49 F1 for 10-shot AutoDiCoT, despite improving accuracy in the former. Defaulting unparsed answers to negative yielded 0.42 F1; averaging three exemplar orderings with extraction yielded 0.36 F1.Source: §6.2.3.3 · pp. 41–42
• 0.45 F1 followed both email anonymization and deduplication, demonstrating sensitivity to seemingly incidental prompt details.Removing the email yielded 0.39 F1; tripling context without duplicating the email yielded 0.47 F1. The full-context-only prompt scored 0.44 F1 and accidentally duplicated the email, whose subsequent deduplication hurt performance.Source: §6.2.3.3 · pp. 40–42
• 0.548 F1 on the test set was achieved by the best DSPy prompt (Khattab et al., 2023), outperforming the human-engineered prompts.The prompt achieved 0.385 precision and 0.952 recall, using 15 exemplars without CoT reasoning and one bootstrapped reasoning demonstration. Optimization explored demonstrations and training exemplars over 16 iterations; the best prompt omitted both the professor’s email and the incorrect explicitness instruction.Source: Figure 6.19 · p. 42, §6.2.3.3 · p. 42

6.2.4 Discussion

• Prompt engineering requires examining errors and collaboration between prompt engineers and domain experts because model behavior can depend unpredictably on prompt details.The discussion also emphasizes sensitivity to the specific LLM and identifies engagement with domain experts as its most important takeaway.Source: §6.2.4 · pp. 42–43
• Combining automated exploration with human prompt engineering and revision was the study’s most successful approach.The authors identify significant promise in automated exploration of the prompting space.Source: §6.2.4 · p. 43

7 Related Work

• Unlike the broad pre-ChatGPT review by Liu et al. (2023b), this survey examines discrete prefix prompting in greater depth.Chen et al. (2023a) review popular prompting techniques, while Chu et al. (2023) focus on Chain-of-Thought reasoning methods. White et al. (2023) and Schmidt et al. (2023) offer prompt-pattern taxonomies, while Santu and Feng (2023) provide a general prompt taxonomy.Source: §7 · p. 44
• Healthcare-focused reviews address prompting use cases, limitations, medical education, and mental health applications.Meskó (2023) and Wang et al. (2023d) discuss healthcare uses and limitations; Heston and Khun (2023) cover medical education. Hua et al. (2024) use a GPT-4-automated approach to review LLMs in mental health.Source: §7 · p. 44
• The survey distinguishes itself through broad coverage of generally applicable prompting techniques, taxonomic organization, terminology standardization, and PRISMA-based review methods.It adopts PRISMA (Page et al., 2021) as its systematic literature-review standard.Source: §7 · p. 44
• Modality-specific reviews cover visual prompting, multimodal interactions, embodied agents, and text-to-image tools for artists.Wang et al. (2023c) review visual prompting and relevant models; Yang et al. (2023e) emphasize qualitative analyses of multimodal prompting. Durante et al. (2024) cover LLM embodied agents, while Ko et al. (2023b) examine artists’ adoption of text-to-image generation.Source: §7 · p. 44
• Software-focused reviews examine prompt engineering, software testing, program repair, and computer science education.Hou et al. (2023) systematically review software-engineering prompting; Wang et al. (2023e) examine LLM-based testing. Zhang et al. (2023a) review ChatGPT prompting performance on software-engineering tasks, and Neagu (2023) covers computer science education.Source: §7 · p. 44
• Adjacent surveys address fairness, hallucination, verifiability, reasoning, augmentation, and linguistic properties of prompts.These include Li et al. (2023j) on fairness, Huang et al. (2023b) on hallucination, and Liu et al. (2023a) on verifiability. Qiao et al. (2022), Mialon et al. (2023), and Leidinger et al. (2023) cover reasoning, augmentation, and prompt linguistics, respectively.Source: §7 · p. 44

8 Conclusions

• Prompting combines linguistic ambiguity, context dependence, and course correction with uncertainty about whether models understand language in ways resembling humans.The authors suggest describing supposedly emergent techniques as discoveries arising from experimentation, analogies to human reasoning, or serendipity.Source: §8 · p. 45
• Over 200 prompting techniques are discussed alongside associated frameworks and safety/security issues within a taxonomy intended to accommodate future methods.Two case studies illustrate model capabilities and practical problem-solving. The authors acknowledge likely gaps and redundancies in the taxonomy.Source: §8 · p. 45
• The survey does not validate technique effectiveness, emphasizing unstandardized evaluation and uncertain transfer across models, problems, and datasets.Even meticulous experiments may have unforeseen shortcomings, and meaning-preserving input changes can affect model outputs.Source: §8 · p. 45
• Practitioners should understand the underlying problem, use representative data and metrics, start with simpler approaches, and remain skeptical of performance claims.Problem understanding should extend beyond input/output behavior and benchmark scores.Source: §8 · p. 45
• Researchers should situate new methods within the taxonomy and include ecologically valid case studies and illustrations.The taxonomy is also intended to clarify relationships between existing techniques for experienced practitioners.Source: §8 · p. 45

A.2.1 Prompting Terms

• The context window is the space of tokens an LLM can process.Source: §A.2.1 · p. 64
• Context length names the maximum length of the context window.Source: §A.2.1 · p. 64
• Priming (Schulhoff, 2022) supplies an initial prompt that sets instructions for the rest of a conversation.The prompt may specify a role or instructions for interacting with the user. Priming can appear in either the system or user prompt.Source: §A.2.1 · p. 64

• Conversational Prompt Engineering takes place within a conversation with a GenAI.Source: §A.2.2 · p. 64
• Users may ask the GenAI to refine its output during that conversation.Source: §A.2.2 · p. 64
• Prompt engineering often involves sending a completely new prompt rather than continuing a conversation.Source: §A.2.2 · p. 64

A.2.3 Fine-Tuning Terms

• The survey avoids the terms Prompt-Based Learning and Prompt Learning because of conflicting usage.These labels often appear in fine-tuning contexts, especially when fine-tuning prompts.Source: §A.2.3 · p. 64
• Prompt-Based Learning (Liu et al., 2023b), also called Prompt Learning (Liu et al., 2023b; Wang et al., 2023d), broadly refers to using prompting-related techniques.The definition is not restricted to fine-tuning.Source: §A.2.3 · p. 64
• Prompt Tuning, also called Prompt Fine-Tuning, directly optimizes prompt weights, usually through gradient-based updates.The term should not refer to discrete prompt engineering.Source: §A.2.3 · p. 64

A.2.4 Orthogonal Prompt Types

• The terminology organizes prompts along high-level dimensions of originator, token representation, and prediction style.These dimensions distinguish who supplies a prompt, what its tokens represent, and where predicted tokens occur.Source: §A.2.4 · p. 64, §A.2.4.1 · p. 64, §A.2.4.2 · p. 64

A.2.4.1 Originator

• User prompts originate from users and are the most common way prompts are delivered in consumer applications.Source: §A.2.4.1 · p. 64
• Assistant prompts are model outputs that become prompts when fed back into the model, such as through conversation history.Source: §A.2.4.1 · p. 64
• System prompts provide high-level instructions for interacting with users, but not all models support them.Source: §A.2.4.1 · p. 64

A.2.4.2 Hard vs Soft Prompts

• Hard, or discrete, prompts contain only tokens that directly correspond to words in the model’s vocabulary.Source: §A.2.4.2 · p. 64
• Soft prompts can contain tokens without vocabulary-word equivalents (Lester et al., 2021; Wang et al., 2023c).When full-model weight updates are prohibitively expensive, the model can remain frozen while gradients flow through prompt tokens.Source: §A.2.4.2 · p. 64

A.2.4.3 Prediction Styles

• Text prediction styles distinguish cloze prompts’ fill-in slots from prefix prompts’ end-position predictions (Liu et al., 2023b).Cloze slots usually occur within the prompt and are typical of earlier models such as BERT (Chu and Lin, 2023). Prefix prediction is typical of modern GPT-style models; non-text prediction styles are excluded from this discussion.Source: §A.2.4.3 · pp. 64–65

A.3 Datasheet

• The datasheet provides additional information about the paper’s associated dataset.Source: §A.3 · p. 66
• The associated dataset is hosted on HuggingFace.Source: §A.3 · p. 66
• The paper presents its dataset documentation as a datasheet.Source: §A.3 · p. 66

A.3.2 Composition

• 1,565 research papers comprise the unlabeled PDF dataset, with possible residual duplicates despite automatic deduplication.Manual reviews mitigated irrelevant inclusions and missed relevant papers from semi-automated collection. Some prompt-injection papers may contain racist or sexist content.Source: §A.3.2 · p. 66

How was the data associated with each instance acquired? / A.3.4 Preprocessing/ Cleaning/ Labeling

• Papers came from Arxiv, Semantic Scholar, and ACL, with scripts querying the Arxiv and Semantic Scholar APIs.Cleaning removed duplicates and used manual and semi-automated relevance reviews; preprocessing software is available in the GitHub repository. No ethical review was conducted.Source: How was the data associated with each… · p. 67, §A.3.4 · p. 67

Has the dataset been used for any tasks already?

• All collected papers were written in English, potentially excluding relevant papers for which translations were unavailable.Source: Has the dataset been used for any… · p. 67

A.5 Prompt for Systematic Literature Review

• The relevance-screening prompt restricts inclusion to hard prefix prompting and excludes papers focused on model training rather than post-training prompting.Titles and abstracts receive one of five relevance ratings, with supporting abstract text requested. The system prompt specifies JSON containing reasoning and rating fields.Source: §A.5 · p. 70

A.7.2 Getting a Label

• Switching to GPT-4-32K produced one-word responses and correctly classified one accept example and one reject example.Earlier yes/no wording elicited labels, but misclassification and supportive responses persisted, including with 10-shot prompting.Source: §A.7.2 · p. 72

A.7.3 Varying Prompting Techniques

• 10-shot prompting with context showed improvement, but adding ten examples before a reasoning exemplar produced poor results.A prompt containing example reasoning performed well outside the first 20 instances in informal testing. Removing an included email sharply reduced performance.Source: §A.7.3 · pp. 72–73
• The model inferred entrapment from distressing circumstances despite acknowledging that the post did not explicitly express feeling trapped.Step-by-step reasoning supported an entrapment label, and an instruction requiring explicit statements still yielded “Yes.”Source: §A.7.3 · pp. 72–73

A.7.3.1 AutoCoT •

• 0.5 F1 was exceeded in tests using AutoCoT-generated data, but output-extraction failures persisted.Examples were generated by requesting explanations and correcting disagreements. Extraction failures increased from about 11% to about 16% after abbreviated field names were expanded.Source: §A.7.3.1 · p. 74

A.7.3.2 Developing Answer Extraction •

• Answer extraction offered only slight gains, including a small accuracy increase when applied to -1 outputs from the best-performing 10-QRA-shot prompt.Restricting extraction to outputs labeled -1 helped slightly; broader improvements were not observed for the best-performing prompt.Source: §A.7.3.2 · p. 74

A.7.3.3 Iterating on Email

• Deduplicating emails worsened results, while triplication and exemplar reordering with extraction did not help.Uncertain predictions often assigned label 1 where label 0 was correct, so attempts to recover these predictions were unhelpful.Source: §A.7.3.3 · p. 74

A.8 Formally Defining a Prompt

• The formalization treats a prompt template as a function mapping inputs to model-conditioning prompts, with training examples added in few-shot settings.The section introduces these formalisms because inconsistent definitions make a single mathematical definition of a prompt difficult.Source: §A.8 · p. 75
• Prompt optimization evaluates templates against dataset-based scoring functions, while answer engineering transforms raw model outputs for comparison with ground truth.Scoring may be reference-free when labels are unavailable. The deliberately simple framework could be extended to chain-of-thought prompting and retrieval systems.Source: §A.8 · pp. 75–76

A.9 In-Context Learning Definitions Disambiguation

• The survey adopts Brown et al. (2020)’s broad ICL definition, encompassing zero-shot, one-shot, and few-shot settings without requiring new-task learning.Brown et al. (2020) distinguish the inference-time inner loop from meta-learning’s inner/outer-loop structure. The survey notes that its definition differs from Dong et al. (2023)’s formal definition.Source: §A.9 · p. 78

A.10 Contributions

• The contributions record documents cross-section reviews alongside multilingual leadership, meta-analysis visualizations, experiments, data collection, and codebase management.Dayeon Ki led multilingual prompting; Aayush Gupta contributed meta-analysis visualizations; co-lead author Michael Ilie managed code, experiments, and data collection.Source: §A.10 · p. 79, Multilingual Prompting and Meta Analysis · p. 80