Source-linked AI summary
A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications
Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, Aman Chadha
TL;DR
Prompt engineering lacks a systematic organization of its diverse methods despite its growing role in adapting LLMs and VLMs without parameter updates. This survey categorizes 41 techniques by application area and analyzes their methods, applications, models, datasets, strengths, and limitations. Its synthesis includes reasoning, efficiency, and emotional-prompting results while identifying open challenges including bias, factual inaccuracies, interpretability gaps, and rationale-quality limitations.
Problem
The diverse prompt engineering methods and techniques lack systematic organization and understanding, limiting a structured view of this rapidly developing field.
Method
The survey categorizes 41 prompt engineering techniques by targeted functionality and reviews their applications, models, datasets, strengths, limitations, taxonomy, and evaluation metrics.
Results
The survey reports gains across reviewed approaches, including 3.41% and 5.08% improvements over CoT with GoT on GSM8K, plus 8.00% instruction-induction and 115% BIG-Bench improvements with EmotionPrompt.
Takeaways & Limitations
The structured synthesis supports deeper understanding of prompt engineering and facilitates future research by illuminating open challenges and opportunities.
Takeaways & Limitations
The survey notes that ECHO remained behind GPT-3.5, attributing the gap to differences in reasoning-rationale quality.
Abstract
from arXiv · showhide
Prompt engineering has emerged as an indispensable technique for extending the capabilities of large language models (LLMs) and vision-language models (VLMs). This approach leverages task-specific instructions, known as prompts, to enhance model efficacy without modifying the core model parameters. Rather than updating the model parameters, prompts allow seamless integration of pre-trained models into downstream tasks by eliciting desired model behaviors solely based on the given prompt. Prompts can be natural language instructions that provide context to guide the model or learned vector representations that activate relevant knowledge. This burgeoning field has enabled success across various applications, from question-answering to commonsense reasoning. However, there remains a lack of systematic organization and understanding of the diverse prompt engineering methods and techniques. This survey paper addresses the gap by providing a structured overview of recent advancements in prompt engineering, categorized by application area. For each prompting approach, we provide a summary detailing the prompting methodology, its applications, the models involved, and the datasets utilized. We also delve into the strengths and limitations of each approach and include a taxonomy diagram and table summarizing datasets, models, and critical points of each prompting technique. This systematic analysis enables a better understanding of this rapidly developing field and facilitates future research by illuminating open challenges and opportunities for prompt engineering.
1 Introduction
Prompt engineering guides LLMs and VLMs with task-specific instructions while leaving model parameters unchanged. The survey reviews this evolving field systematically, covering methods, applications, models, datasets, strengths, and limitations.
- Prompt engineering uses task-specific instructions to guide LLM and VLM outputs without altering model parameters.
- Its adaptability supports diverse tasks and domains without relying on traditional retraining or extensive fine-tuning.
- The field spans foundational zero-shot and few-shot prompting alongside more intricate approaches such as chain-of-code prompting.
- The survey systematically examines prompting methods, applications, language models, experimental datasets, strengths, limitations, and comparative efficacy.
- It provides a taxonomy diagram and a table summarizing datasets, models, and evaluation metrics across applications including generation, question answering, coding, and reasoning.
2 Prompt Engineering
The survey organizes prompt engineering techniques by application area, ranging from zero-shot task transfer to structured, symbolic, graph-based, and efficiency-oriented reasoning methods. These approaches adapt demonstrations, attention, logic, search, and output length to address different task demands.
- 2.1 New Tasks Without Extensive Training: Zero-shot prompting performs new tasks from task descriptions and pretrained knowledge without labeled input-output training examples.
- 2.1 New Tasks Without Extensive Training: Few-shot prompting supplies a small number of examples to induce task understanding, but example selection and added tokens can affect behavior and cost.
- 2.2 Reasoning and Logic: Chain-of-Thought prompting elicits coherent, step-by-step reasoning for complex tasks, while Auto-CoT automates reasoning-chain generation through diverse sampling.
- 2.2 Reasoning and Logic: Self-consistency samples diverse reasoning chains and selects the most consistent final answer instead of relying on greedy decoding.
- 2.2 Reasoning and Logic: LogiCoT verifies each generated reasoning step using reductio ad absurdum and gives targeted feedback for revising incorrect steps.
- 2.2 Reasoning and Logic: Chain-of-Symbol replaces natural-language descriptions with condensed symbols for clearer spatial reasoning, reducing prompt tokens by up to 65.8%.
- 2.2 Reasoning and Logic: Tree-of-Thoughts searches and evaluates a tree of intermediate thoughts, while Graph-of-Thoughts models reasoning as a directed graph with dynamic transformations.ToT achieved 74% versus CoT’s 4% on Game of 24; GoT improved accuracy over CoT by 3.41% with T5-base and 5.08% with T5-large on GSM8K.
- 2.2 Reasoning and Logic: Logic-of-Thought extracts and extends propositions with formal logical laws before translating them back into prompts, improving compatibility with CoT, Self-Consistency, and ToT.
2.3 Reduce Hallucination
The survey describes prompting methods that augment, coordinate, verify, or assess information to improve reasoning and reduce unreliable outputs. These approaches address limitations involving static knowledge, hallucinations, and unreliable retrieval.
- Retrieval Augmented Generation integrates information retrieval into prompting by analyzing user input, crafting a query, and retrieving external knowledge.
- ReAct interleaves reasoning traces with task-specific actions so models can induce, track, and update plans while handling exceptions.
- Chain-of-Verification generates a baseline response, plans verification questions, answers them independently, and revises the response to reduce hallucinations.
- Chain-of-Note addresses unreliable retrieved information by helping retrieval-augmented models assess knowledge adequacy and respond with unknown when information is insufficient.
- Chain-of-Knowledge decomposes complex tasks into coordinated reasoning preparation and dynamic knowledge adaptation to address fixed knowledge sources and weak query generation.
2.4 User Interface
Active Prompting adapts reasoning prompts to diverse tasks by selecting task-specific examples rather than relying on a fixed set of human-annotated exemplars.
- Active Prompting determines impactful questions and uses task-specific chain-of-thought examples to adapt LLMs to diverse reasoning tasks.
2.5 Fine-Tuning and Optimization
Automatic Prompt Engineer (APE) automates instruction generation and selection, adapting prompts to specific tasks and contexts instead of relying solely on static, hand-designed prompts.
- 2.5 Fine-Tuning and Optimization: APE analyzes user input, generates candidate instructions, and uses reinforcement learning to select the most impactful prompt.The method dynamically adapts prompt selection to different contexts.
- 2.5 Fine-Tuning and Optimization: APE exceeded human-authored prompts on 19 of 24 BIG-Bench and Chain-of-Thought reasoning tasks.The reported evaluations indicate stronger performance than human-authored prompts in most tested tasks.
- 2.5 Fine-Tuning and Optimization: APE was designed to reduce the labor associated with crafting effective prompts for LLMs.It addresses the traditional reliance on expert annotators to create prompts.
2.6 Knowledge-Based Reasoning and Generation
Automatic Reasoning and Tool-use (ART) addresses limited LLM reasoning and external-tool use by automating multi-step reasoning and integrating specialized tools.
- 2.6 Knowledge-Based Reasoning and Generation: ART enables LLMs to reason through multi-step processes while integrating external expertise and tools.External tools provide specialized knowledge and computations for complex tasks.
- 2.6 Knowledge-Based Reasoning and Generation: ART uses structured programs to automate reasoning steps, reducing the need for laborious hand-crafted prompts.Its dynamic tool integration extends prompting beyond simple text generation.
- 2.6 Knowledge-Based Reasoning and Generation: ART supports applications including scientific research, data analysis, and decision-making support.The passage links these applications to the method’s integration of external tools and real-world information.
2.7 Improving Consistency and Coherence
Contrastive Chain-of-Thought (CCoT) prompting augments standard reasoning demonstrations with valid and invalid examples, helping LLMs learn from both correct and incorrect reasoning paths.
- 2.7 Improving Consistency and Coherence: CCoT provides valid and invalid reasoning demonstrations alongside the original prompt.This dual-perspective design explicitly includes mistakes and alternative reasoning paths.
- 2.7 Improving Consistency and Coherence: CCoT improved strategic and mathematical reasoning evaluations by 4-16% compared with traditional CoT.The reported tests used reasoning benchmarks including SQuAD and COPA.
- 2.7 Improving Consistency and Coherence: Integrating CCoT with self-consistency techniques produced an additional improvement of approximately 5%.This gain was reported beyond the improvements over traditional CoT.
- 2.7 Improving Consistency and Coherence: Open questions include automatically generating contrasting demonstrations and extending CCoT beyond reasoning-focused NLP tasks.The passage identifies both demonstration construction and broader task applicability as unresolved issues.
2.8 Managing Emotions and Tone
EmotionPrompt appends emotional stimulus sentences to task prompts to investigate and improve LLM handling of psychological and emotional cues.
- 2.8 Managing Emotions and Tone: EmotionPrompt appends 11 emotional stimulus sentences to prompts to enhance LLM emotional intelligence.The design draws on psychological research concerning language’s effects on human performance.
- 2.8 Managing Emotions and Tone: EmotionPrompt produced an 8.00% relative improvement in instruction induction and a 115% boost on BIG-Bench tasks.These results were reported across the study’s evaluated tasks.
- 2.8 Managing Emotions and Tone: An evaluation with 106 participants found an average 10.9% improvement in performance, truthfulness, and responsibility for generative tasks.The comparison was against standard prompts.
2.9 Code Generation and Execution
Code-focused prompting methods address weaknesses in complex reasoning, arithmetic, and code generation by structuring intermediate reasoning or computation. These approaches include scratchpads, external interpreters, structured reasoning, and code-based reasoning.
- Scratchpad Prompting: Scratchpad Prompting lets models generate arbitrary intermediate tokens before the final answer for complex, multi-step algorithmic calculations.It is a task-design approach that avoids modifying model parameters and reaches a 46.8% success rate on MBPP-aug.
- Cross-method synthesis: Together, these methods use intermediate representations—tokens, programs, structured steps, or pseudocode—to make difficult reasoning and execution tasks more manageable.The supplied descriptions cover complementary strategies for algorithmic calculation, numerical reasoning, code generation, and symbolic reasoning.
- Program of Thoughts (PoT) Prompting: Program of Thoughts uses external language interpreters to handle computation steps and reduce arithmetic and equation-solving errors.The method separates numerical computation from the language model’s reasoning process.
- Structured Chain-of-Thought (SCoT) Prompting: Structured Chain-of-Thought adapts intermediate reasoning to code generation, where natural-language CoT has lower accuracy than in language-generation settings.Its motivation is to tailor the reasoning format specifically to programming tasks.
- Chain-of-Code (CoC) Prompting: Chain-of-Code extends CoT by using codewriting for both logic and semantic tasks, including pseudocode for semantic subtasks.The approach is effective with both large and small models and promotes a “think in code” strategy for reasoning questions.
2.10 Optimization and Efficiency
Optimization by Prompting uses language models as natural-language optimizers that iteratively generate solutions and prompts for varied tasks. Its experiments show that optimized prompts can outperform human-designed prompts substantially.
- Optimization by Prompting (OPRO): Optimization by Prompting iteratively generates solutions from natural-language problem descriptions, allowing rapid task adaptation and customization.The approach treats the LLM as an optimizer rather than relying exclusively on traditional optimization procedures.
- Applications: OPRO is evaluated on linear regression, the traveling salesman problem, and prompt optimization for natural-language-processing accuracy.The experiments also examine how optimizing prompts on a small training set transfers to test performance.
2.11 Understanding User Intent
Prompting methods for understanding user intent improve responses by reformulating questions or abstracting from specific instances before reasoning. The section also situates prompting techniques within a broader tabular organization of applications, models, datasets, and metrics.
- Rephrase and Respond (RaR) Prompting: Rephrase and Respond lets an LLM rephrase and expand a question within a prompt to improve comprehension and response accuracy.A two-step variant uses separate rephrasing and response LLMs and reports substantial performance enhancements across various settings.
- Take a Step Back Prompting: Take a Step Back uses abstraction followed by reasoning, extracting high-level concepts and fundamental principles from specific instances.The technique is designed for complex multi-step reasoning with advanced models such as PaLM-2L.
- Survey organization: The survey table organizes prevalent prompting techniques by application, prompt acquisition, prompt turn, language model, dataset, and metrics.This organization provides a compact comparison framework for the techniques reviewed in the survey.
3 Conclusion
The survey organizes 41 prompt engineering techniques by targeted functionality and reviews their applications, models, datasets, strengths, and limitations. It concludes that prompt engineering has achieved notable progress while retaining unresolved technical and ethical challenges.
- Survey contribution: The survey categorizes 41 prompt engineering techniques according to their targeted functionalities and summarizes their applications, models, and datasets.It also includes a diagram and table highlighting important points for comparison.
- Survey contribution: The analysis discusses strengths and limitations across prompting approaches, providing a systematic overview intended to support further research.The conclusion presents the survey as a foundational resource for the evolving prompt-engineering field.
- Open challenges: Prompt engineering continues to face biases, factual inaccuracies, and interpretability gaps despite its reported successes.The paper identifies further investigation and mitigation strategies as necessary responses to these challenges.
- Future directions: The paper highlights meta-learning and hybrid prompting architectures as emerging trends, while emphasizing ethical considerations in responsible development and deployment.These future directions and safeguards are presented as part of prompt engineering’s continued evolution.