PaperlayerView exact revision on arXiv

Source-linked AI summary

Unleashing the potential of prompt engineering for large language models

Banghao Chen, Zhaofeng Zhang, Nicolas Langrené, Shengxin Zhu

arXiv:2310.14735v6cs.CLcs.AI

TL;DR

Prompt engineering addresses the challenge of manually crafting effective prompts and the broader problem of inaccurate or unsupported LLM outputs. This review synthesizes prompt-optimization methods, retrieval augmentation, applications, and security considerations, finding reported gains of up to 31% over initial prompts for ProTeGi while emphasizing secure deployment boundaries.

  • Problem

    Crafting effective prompts requires extensive manual effort and expertise, while LLMs can generate fluent but unreal or inaccurate information.

  • Method

    The paper reviews prompt-engineering methods and examines data-driven prompt optimization, retrieval augmentation, applications, and security practices.

  • Results

    ProTeGi improved prompt performance by up to 31% over initial prompts across sentiment analysis, fake news detection, and LLM jailbreak detection.

  • Takeaways & Limitations

    Prompt engineering provides a scalable approach to improving LLM outputs and supports applications spanning education, content creation, programming, reasoning, and security.

  • Takeaways & Limitations

    Secure deployment remains necessary because prompt engineering can introduce risks from adversarial prompts, while Golden CoT depends on ground-truth chain-of-thought solutions.

Abstract

from arXiv · show

This comprehensive review delves into the pivotal role of prompt engineering in unleashing the capabilities of Large Language Models (LLMs). The development of Artificial Intelligence (AI), from its inception in the 1950s to the emergence of advanced neural networks and deep learning architectures, has made a breakthrough in LLMs, with models such as GPT-4o and Claude-3, and in Vision-Language Models (VLMs), with models such as CLIP and ALIGN. Prompt engineering is the process of structuring inputs, which has emerged as a crucial technique to maximize the utility and accuracy of these models. This paper explores both foundational and advanced methodologies of prompt engineering, including techniques such as self-consistency, chain-of-thought, and generated knowledge, which significantly enhance model performance. Additionally, it examines the prompt method of VLMs through innovative approaches such as Context Optimization (CoOp), Conditional Context Optimization (CoCoOp), and Multimodal Prompt Learning (MaPLe). Critical to this discussion is the aspect of AI security, particularly adversarial attacks that exploit vulnerabilities in prompt engineering. Strategies to mitigate these risks and enhance model robustness are thoroughly reviewed. The evaluation of prompt methods is also addressed through both subjective and objective metrics, ensuring a robust analysis of their efficacy. This review also reflects the essential role of prompt engineering in advancing AI capabilities, providing a structured framework for future research and application.

1 Introduction

Prompt engineering has evolved into a structured research domain alongside advances in LLMs, whose outputs are strongly influenced by prompt design. This review surveys foundational and advanced prompt methods, VLM approaches, evaluations, applications, and security.

  • LLM foundations: LLMs use transformer architectures and self-supervised next-word prediction over extensive web-derived datasets to generate contextually relevant responses.Their outputs reflect learned statistical associations between words and phrases.
  • Prompt engineering: Prompt structure and content can produce significant output differences, making systematic prompt design important for model behavior.Changes include prompt length, instance arrangement, phrasing, illustrations, and directives.
  • Historical development: Prompt engineering developed from structured inputs in the 1950s to contemporary methods such as chain-of-thought and self-consistency prompting.The review focuses primarily on techniques emerging after 2017.
  • Review scope: The paper reviews foundational techniques, advanced reasoning methods, VLM prompt methods, evaluation approaches, and applications across multiple domains.Covered methods include clear instructions, role-prompting, chain-of-thought, self-consistency, CoOp, CoCoOp, and MaPLe.

2 Basics of prompt engineering

Basic prompt engineering methods improve model guidance by specifying instructions, roles, examples, formatting, and generation settings. The section presents these methods as practical ways to influence response precision, relevance, diversity, and task performance.

  • 2.1 Model introduction: GPT-4: GPT-4 examples in this section use a transformer model fine-tuned with reinforcement learning from human feedback.GPT-4 processes tokenized inputs through attention-based transformer layers before decoding text.
  • 2.1 Model introduction: GPT-4: Temperature controls output randomness while top-k sampling limits generation to the k most probable tokens.Higher temperature produces more random outputs, whereas lower temperature makes outputs more deterministic.
  • 2.2 Giving instructions: Basic instructions without supplementary description can yield excessively general outputs, whereas comprehensive descriptions elicit more precise and relevant responses.The method is also called re-reading.
  • 2.3 Be clear and precise: Clear and precise prompts reduce ambiguity and guide models toward outputs aligned with specific scenario requirements.Detailed prompts reduce model uncertainty compared with broad or undetailed prompts.
  • 2.4 Role-prompting: Role-prompting assigns the model a specific role, such as a historian or knowledgeable expert, to guide the response toward the desired output.The paper presents role assignment as a fundamental prompt method.
  • 2.5 Use of triple quotes to separate: Triple quotes separate prompt components or encapsulate multiline strings, helping models interpret instructions in complex prompts.They are particularly useful when prompts contain multiple components or embedded quotation marks.
  • 2.6 Try several times: Resampling runs the model several times with the same prompt and selects the best output to address nondeterministic response variability.This can increase the chance of obtaining a high-quality response.
  • 2.7 One-shot or few-shot prompting: One-shot prompting supplies one example, few-shot prompting supplies multiple examples, and zero-shot prompting supplies no task-specific examples.One-shot may suit simple tasks or highly capable models, while few-shot can provide added guidance for complex tasks or less capable models; examples can sometimes hurt performance.

3 Advanced methodologies

Advanced prompt-engineering methods address the remaining accuracy challenges of LLMs on complex analysis and reasoning tasks. The review covers chain-of-thought, zero-shot chain-of-thought, golden chain-of-thought, and self-consistency approaches.

  • Advanced prompting methods target improved specificity, accuracy, and quality for complex analysis and reasoning tasks.
  • 3.1 Chain-of-thought: Chain-of-thought prompting guides responses with intermediate reasoning steps, improving accuracy on logical reasoning tasks and making conclusions easier to understand.
  • 3.1.1 Zero-shot chain-of-thought: Zero-shot chain-of-thought prompts can elicit sequential reasoning without task examples, with “Let’s think step by step” producing more precise, coherent, and comprehensive answers.
  • 3.1.2 Golden chain-of-thought: Golden chain-of-thought incorporates ground-truth reasoning solutions into prompts and achieved an 83% solve rate, but its contribution is limited by that requirement.
  • 3.2 Self-consistency: Self-consistency samples diverse reasoning paths and selects the most consistent answer, improving outcomes in arithmetic, commonsense, and symbolic reasoning tasks.

3.3 Generated knowledge

Generated knowledge supplies additional context before a model produces its final response, while least-to-most prompting decomposes complex tasks into sequential subproblems. The reviewed evidence describes improved logical coherence and generalization to more complex challenges.

  • 3.3 Generated knowledge: Generated knowledge prompts an LLM to produce potentially useful information about a question before generating the final response.
  • 3.3 Generated knowledge: For commonsense reasoning, generated information adds context that may be absent from the initial prompt.
  • 3.3 Generated knowledge: In the entrance example, generating relevant information first leads to responses that consider factors such as entrance height and show greater logical coherence and comprehensiveness.
  • 3.4 Least-to-most prompting: Least-to-most prompting decomposes a complex problem into simpler subproblems solved sequentially, with each solution supporting the next step.
  • 3.4 Least-to-most prompting: Experiments in symbolic manipulation, compositional generalization, and mathematical reasoning indicate that least-to-most prompting can generalize to challenges more complex than those initially presented.

3.5 Tree of thoughts

Advanced prompting methods structure reasoning beyond linear generation by exploring multiple thought paths, representing dependencies as graphs, or decomposing complex tasks into modular sub-tasks. These approaches support more flexible reasoning and, in DECOMP case studies, improved performance across several task settings.

  • 3.5 Tree of thoughts: Tree of Thoughts explores multiple reasoning paths, including looking ahead, backtracking, and self-evaluation, rather than following a single linear prompt.This structure is intended to make problem solving more interactive and adaptable to complex tasks.
  • 3.5 Tree of thoughts: The streamlined tree-of-thought prompting method lets LLMs assess intermediate cognitive constructs within a single prompt.An example prompt is provided in Figure 11.
  • 3.6 Graph of thoughts: Graph of Thoughts represents LLM-generated information as vertices connected by dependency edges, enabling diverse traversal methods for intricate problems.The framework can interlink autonomous thoughts or solutions into a graph intended to support precise and comprehensive final solutions.
  • 3.6 Graph of thoughts: Graph of Thoughts requires strong subject-matter comprehension and meticulous prompt design to achieve optimal outcomes.Its adaptability is emphasized for intricate issues requiring multifaceted resolutions.
  • 3.7 Decomposed prompting: DECOMP breaks complex tasks into manageable sub-tasks handled by specialized functions, creating a modular and systematic problem-solving process.A decomposer LLM generates a prompting program whose steps direct simpler sub-queries to auxiliary sub-task functions.
  • 3.7 Decomposed prompting: DECOMP supports independently optimized handlers, error-correcting components, hierarchical or recursive decompositions, and handler reuse across tasks.Compared with Least-to-Most Prompting’s linear progression, DECOMP allows non-linear and recursive decomposition with replaceable sub-task handlers.
  • 3.7 Decomposed prompting: DECOMP outperformed Chain-of-Thought across k-th letter concatenation, list reversal, long-context QA, open-domain QA, and Math QA case studies.The reported gains came from further decomposition, recursive handling, symbolic retrieval, or post-processing that corrected frequent formatting errors.

3.8 Active prompt

Active prompting selects uncertain questions for human annotation and uses the resulting reasoning exemplars during inference. The review presents this process as improving task adaptation and performance while reducing human engineering effort.

  • 3.8 Active prompt: Active prompting generates multiple predictions, estimates uncertainty, and prioritizes the most informative questions for human annotation.Uncertainty can be measured using disagreement, entropy, or variance.
  • 3.8 Active prompt: Annotated chain-of-thought reasoning and answers are used as exemplars to guide the LLM through complex reasoning during inference.The process combines uncertainty estimation, collection and selection, human annotation, and inference.
  • 3.8 Active prompt: Active prompting offers efficient task adaptation and significant performance improvements across various reasoning domains while focusing human effort on uncertain questions.The review describes this focus as reducing human engineering effort and maximizing the use of human expertise.

3.9 Prompt pattern catalog

Prompt engineering is organized through reusable pattern catalogs and automated optimization methods that reduce manual effort while adapting prompts to tasks and models.

  • 3.9 Prompt pattern catalog: A prompt pattern catalog standardizes reusable templates to improve consistency and model interaction quality.Its documented patterns are intended for systematic application across tasks and domains.
  • 3.9 Prompt pattern catalog: The catalog groups 16 prompt patterns into Input Semantics, Output Customization, Error Identification, Prompt Improvement, and Interaction.Each pattern records its classification, intent, motivation, structure, example implementation, and practical consequences.
  • 3.9 Prompt pattern catalog: Combining patterns supports more complex prompting strategies by integrating multiple structured solutions.The approach is illustrated through detailed examples and practical implementations.
  • 3.10 Prompt optimization: Prompt optimization addresses the manual effort and expertise required to craft effective prompts by systematically refining their accuracy and relevance.The goal is to reduce manual trial and error.
  • 3.10 Prompt optimization: Optimization methods include textual-gradient, black-box, and model-adaptive approaches, whose suitability depends on task requirements, model complexity, and available resources.Textual-gradient methods are directed, black-box methods are broadly applicable, and model-adaptive methods customize prompts for specific models.
  • 3.10.1 Prompt optimization with textual gradients: ProTeGi generates natural-language descriptions of prompt flaws and uses them to iteratively refine prompts without accessing model internals.It adapts gradient-descent ideas to discrete natural-language prompts.
  • 3.10.1 Prompt optimization with textual gradients: Up to 31% gains over initial prompts were reported for ProTeGi across sentiment analysis, fake-news detection, and LLM jailbreak detection.The method also reportedly surpassed existing optimization methods in efficiency and accuracy.
  • 3.10.2 Black-box prompt optimization: BPO improves alignment across GPT-3.5 and LLaMA-2 and reportedly outperforms RLHF and DPO independently or in combination.Its model-agnostic operation does not require access to model internals, while prompt changes remain observable.

3.11 Retrieval augmentation

Retrieval augmentation addresses hallucinations by incorporating current external knowledge into model inputs, targeting inaccurate outputs that lack factual support.

  • 3.11 Retrieval augmentation: Hallucinations are unreal or inaccurate generated information that may remain grammatically correct while conflicting with facts or lacking real-world support.The review identifies hallucination reduction as a major direction in prompt engineering.
  • 3.11 Retrieval augmentation: Retrieval augmentation incorporates up-to-date external knowledge into model inputs to reduce hallucinations and improve prompt effectiveness.The approach retrieves pertinent facts from external sources and augments the model’s context.

3.12 Reasoning and active interaction

Reasoning-and-action methods extend prompting with multistep reasoning and external tools, supporting more coherent, accurate, and current responses for complex tasks.

  • 3.12 Reasoning and active interaction: ART combines automatic chain-of-thought prompting with specialized external tools such as calculators and databases.It guides models through multistep reasoning while adding tool-based capabilities.
  • 3.12 Reasoning and active interaction: ART can structure complex tasks into manageable steps and select tools for stages requiring precise calculation, updated information, or specialized processing.A calculator can perform computations after the model outlines a solution path.
  • 3.12 Reasoning and active interaction: ART has shown promising results in complex problem spaces and dynamic information-processing tasks such as automated customer service.The cited work links its use to more effective task navigation.
  • 3.12 Reasoning and active interaction: The review presents ART as relevant to robust AI systems that combine cognitive and computational skills for technical problems.Reported findings emphasize accuracy and reliability in AI applications.
  • 3.12.2 ReAct framework: ReAct prompts models to produce reasoning traces and task-specific actions, including database queries, calculations, and software interactions.The framework sequences contemplation, planning, and execution.
  • 3.12.2 ReAct framework: ReAct can combine reasoning with current external data and calculations for more robust outcomes than static prompts in financial analysis.The model first outlines portfolio-evaluation steps, then gathers market data and performs calculations.
  • 3.12.2 ReAct framework: ReAct supports more informed decisions in finance, healthcare, and legal analysis by using up-to-date, relevant information.The cited passages associate tool interaction with fewer errors from outdated or incomplete data.
  • 3.12.2 ReAct framework: Implementing ReAct requires detailed task and tool knowledge to guide transitions between reasoning and action.Prompt development for both processes is explicitly described as nontrivial.

4 Methodologies for multimodal large models

VLM prompt methods adapt pretrained multimodal models through zero- or few-shot prompting, learned context vectors, and joint vision-language optimization.

  • 4 Methodologies for multimodal large models: VLMs combine visual and linguistic information for tasks including image description and visual question answering.Their multimodal architecture connects image understanding with text generation.
  • 4 Methodologies for multimodal large models: Sophisticated prompt designs address data alignment, modality integration, and context preservation in multimodal tasks.The review highlights CoOp and MaPLe as methods for these challenges.
  • 4 Methodologies for multimodal large models: Zero-shot and few-shot prompting let VLMs perform tasks with minimal or no task-specific training data by relying on pretrained knowledge and examples.Studies examine question templates, image captions, and chain-of-thought reasoning for visual question answering.
  • 4 Methodologies for multimodal large models: Continuous prompt vectors are learned during training to adapt frozen VLMs such as CLIP to complex video-understanding tasks.Only a few parameters are trained, while competitive benchmark performance is reported.
  • 4.3 Context optimization: CoOp embeds learnable context vectors into pretrained VLMs and optimizes them for downstream tasks instead of relying on extensive manual prompt engineering.The vectors are fine-tuned to minimize classification loss.
  • 4.3 Context optimization: CoOp significantly outperforms traditional models in image recognition and VQA, with context ensembling further improving robustness and generalization.The cited work reports gains from combining multiple context vectors.
  • 4.4 Conditional prompt learning: CoCoOp generates image-conditional prompt vectors through a lightweight network while keeping pretrained model parameters unchanged.This enables adaptation to unseen images and questions without fine-tuning the pretrained model.
  • 4.4 Conditional prompt learning: CoCoOp improves caption accuracy and generalization to novel scenes, addressing limitations of static prompt methods such as CoOp.The review also reports stronger robustness in VQA, image classification, and other applications.

5 Assessing the efficacy of prompt methods

Prompt-method evaluation combines human judgment, automatic metrics, benchmarks, and cross-method comparisons because no single measure captures effectiveness across applications. Recent evidence also shows that prompting gains depend strongly on model, task, and prompting regime.

  • 5.1 Subjective and objective evaluations: Subjective assessments align more closely with human intuition but cost more time and money, whereas objective evaluations are faster and cheaper.
  • 5.1.2 Objective evaluations: Automated metrics may diverge from human judgments, so they require caution and should be selected according to the application.
  • 5.1.1 Subjective evaluations: Subjective evaluation uses human judgments of generated content, commonly assessing fluency, accuracy, novelty, and relevance.
  • 5.1.2 Objective evaluations: Objective evaluation uses automated metrics or benchmark tests to quantify the effectiveness of prompt methods.
  • 5.1.2 Objective evaluations: Benchmarks span mathematical word problems, question answering, language understanding, summarization, and multimodal tasks, enabling standardized comparisons.
  • 5.2 Comparing different prompt methods: Prompt methods can be compared using prediction accuracy, proof accuracy, economic cost, evaluation speed, domain coverage, and human ratings.
  • 5.2 Comparing different prompt methods: InstructEval reports that generic or omitted prompts often perform best in few-shot settings, while expert-written task-specific prompts can substantially help in zero-shot settings.
  • 5.2 Comparing different prompt methods: Automated prompt-generation methods show inconsistent performance across models and task types, indicating limited generalization.

6 Applications improved by prompt engineering

Prompt engineering is presented as a way to improve practical LLM applications across education, content creation, programming, reasoning, and dataset generation. The reviewed examples use tailored prompts, iterative procedures, decomposition, and contextual retrieval to shape outputs.

  • 6.1 Assessment in teaching and learning: Prompt engineering supports personalized educational environments by adapting learning pace, style, assessments, and content to individual students.
  • 6.1 Assessment in teaching and learning: Well-designed prompts can support preliminary automated grading, provide instant feedback, and reveal learning patterns from assessment data.
  • 6.2 Content creation and editing: Content-creation systems use prompting for cross-lingual stories and recursive revision of plans containing settings, characters, and outlines.
  • 6.3 Computer programming: Self-debugging prompts combine feedback, unit tests, and code explanations, while multi-turn programming decomposes implementation into multiple natural-language steps.
  • 6.3 Computer programming: Repo-Level Prompt Generator dynamically retrieves repository context and uses a proposal classifier to construct prompts for code auto-completion.
  • 6.4 Reasoning tasks: Few-shot reasoning traces, self-talk, and chain-of-thought prompting encourage verbalized reasoning steps for word-based mathematical problems.
  • 6.4 Reasoning tasks: Zero-shot chain-of-thought prompting significantly improves GPT-3 and PaLM performance over conventional zero-shot and few-shot prompting on MultiArith and GSM8K.
  • 6.5 Dataset generation: LLMs can generate unlabeled, training, assisted-training, and supplementary synthetic data for smaller or domain-specific classification models.

7 LLMs security

Prompt engineering improves LLM functionality while exposing security risks from manipulated inputs, poisoned training data, hidden backdoors, prompt hacking, and model extraction. The review therefore emphasizes continuous research, testing, and robust design before deployment in critical applications.

  • Prompt engineering can help identify and mitigate LLM vulnerabilities while serving as part of a broader security framework.
  • Security failures are especially consequential in healthcare, finance, and cybersecurity, where attacks may expose sensitive information or disrupt essential services.
  • Adversarial attacks subtly alter prompts or inputs to induce unintended, harmful, incorrect, or biased outputs.
  • Adversarial demonstration attacks can manipulate erroneous predictions through small input changes rather than model randomness.
  • Data poisoning injects malicious training data that distorts learning and can produce erroneous deployed outputs, while backdoors embed malicious patterns that activate through specific triggers.
  • Backdoor attacks may remain dormant on clean inputs but produce harmful or incorrect outputs when an implanted trigger is present.
  • Carefully designed prompts can test susceptibility to adversarial attacks, poisoning, and backdoor activation before secure deployment.
  • Model stealing uses diverse, informative prompts to reconstruct a black-box model’s behavior or extract proprietary knowledge through systematic queries.

8 Prospective methodologies

Future prompt engineering research emphasizes understanding model structure and developing AI agents that combine models, templates, chains, planning, memory, and tool use. These directions aim to better align outputs with user intent while raising new challenges for prompt-based self-regulation.

  • Understanding neural-network layers, attention mechanisms, nodes, and weights could improve how prompts guide model outputs toward user intent.
  • Architectural understanding may reveal why prompts fail and guide improvements to prompting methods.
  • The Causal Transformer exemplifies research intended to delineate causal relationships within data and potentially support more efficient prompt design.
  • A stronger grasp of model architecture could advance explainable AI, trust, and integration across industries.
  • AI agents combine large models with memory, active planning, and tool use to remember information, strategize, and act.
  • The proposed agent evolution progresses through models, prompt templates, chains, agents, and multi-agents.
  • Prompt templates standardize instructions and help align generated outputs with desired outcomes and user instructions.
  • Adaptive agents create a prompt-engineering challenge: designing prompts that steer autonomous behavior toward self-regulation.

9 Conclusion

Prompt engineering is presented as essential for optimizing LLM performance, combining foundational and advanced methods, while CoOp and MaPLe support visual-textual integration in VLMs.

  • Prompt engineering is described as essential for optimizing LLM performance.
  • Clear instructions and role-prompting are identified as foundational methods for enhancing LLM capabilities.
  • Chain-of-thought and self-consistency are identified as advanced methodologies that can significantly enhance LLM capabilities.
  • CoOp and MaPLe are presented as strategies for integrating and optimizing visual and textual data in VLMs.
Loading 2310.14735v6…