PaperlayerView exact revision on arXiv

Source-linked AI summary

Unleashing the potential of prompt engineering for large language models

Banghao Chen, Zhaofeng Zhang, Nicolas Langrené, Shengxin Zhu

arXiv:2310.14735v6cs.CLcs.AI

TL;DR

Effective prompt design remains difficult because it requires substantial manual effort and expertise. This review synthesizes prompting methods, evaluation approaches, applications, and security considerations, finding that prompt engineering enhances model capabilities across diverse applications while requiring secure deployment practices.

  • Problem

    Crafting effective prompts remains a significant barrier because it requires extensive manual effort and expertise.

    Source: §3.10 · p. 17

  • Method

    The review synthesizes foundational and advanced prompting methods, optimization, retrieval augmentation, applications, evaluation, and security practices for language and vision-language models.

    Source: §3.11 · p. 21, §9 · p. 38

  • Results

    Prompt engineering methods enhance model capabilities, with reported effects across diverse applications and gains of up to 31% for ProTeGi over initial prompts.

    Source: §3.10.1 · p. 18, §9 · p. 38

  • Takeaways & Limitations

    Prompt engineering provides a structured framework for improving model performance across education, content creation, programming, and security-related applications.

    Source: §6.1 · p. 30, §9 · p. 38

    Prompt engineering can introduce significant security risks when not managed properly, requiring secure and robust design practices for deployment.

    Source: §7.4 · p. 37

Abstract

from arXiv · show

This comprehensive review delves into the pivotal role of prompt engineering in unleashing the capabilities of Large Language Models (LLMs). The development of Artificial Intelligence (AI), from its inception in the 1950s to the emergence of advanced neural networks and deep learning architectures, has made a breakthrough in LLMs, with models such as GPT-4o and Claude-3, and in Vision-Language Models (VLMs), with models such as CLIP and ALIGN. Prompt engineering is the process of structuring inputs, which has emerged as a crucial technique to maximize the utility and accuracy of these models. This paper explores both foundational and advanced methodologies of prompt engineering, including techniques such as self-consistency, chain-of-thought, and generated knowledge, which significantly enhance model performance. Additionally, it examines the prompt method of VLMs through innovative approaches such as Context Optimization (CoOp), Conditional Context Optimization (CoCoOp), and Multimodal Prompt Learning (MaPLe). Critical to this discussion is the aspect of AI security, particularly adversarial attacks that exploit vulnerabilities in prompt engineering. Strategies to mitigate these risks and enhance model robustness are thoroughly reviewed. The evaluation of prompt methods is also addressed through both subjective and objective metrics, ensuring a robust analysis of their efficacy. This review also reflects the essential role of prompt engineering in advancing AI capabilities, providing a structured framework for future research and application.

1 Introduction

Prompt engineering has become a structured research domain for systematically designing inputs that improve LLM response accuracy, relevance, and coherence. This review surveys foundational and advanced LLM methods, VLM-specific approaches, evaluation, applications, security, and future directions.

Background

  • LLMs and multimodal models have expanded AI capabilities across text, images, audio, information extraction, content creation, image description, and visual question answering.The review primarily focuses on LLMs while incorporating VLMs to examine prompt engineering for multimodal data.Source: §1 · p. 2

Motivation

  • Prompt structure and content can substantially alter model behavior, while well-constructed prompts can help counter hallucinations and improve task performance.Relevant changes include prompt length, instance arrangement, phrasing, illustrations, and directives [20] [27].Source: §1 · pp. 2–3
  • Prompt engineering systematically designs and optimizes inputs to guide LLM responses toward greater accuracy, relevance, and coherence across diverse domains.Its evolution spans structured inputs in the 1950s to contemporary methods such as chain-of-thought prompting and self-consistency prompting.Source: §1 · pp. 2–3

Review scope

  • The review covers foundational methods, advanced techniques including chain-of-thought, self-consistency, and generated knowledge, and VLM methods such as CoOp, CoCoOp, and MaPLe.These methods are presented alongside evaluations of prompt efficacy through subjective and objective metrics.Source: §1 · pp. 3–4
  • The paper further examines prompt engineering applications, security vulnerabilities and mitigations, and future directions involving AI model structures and AI agents.Applications include education, content creation, computer programming, and reasoning tasks; security strategies include adversarial training.Source: §1 · p. 4

2 Basics of prompt engineering

This section presents foundational prompt-engineering practices for eliciting more precise, relevant, and effective LLM outputs, while explaining GPT-4’s processing and generation settings. It covers instruction design, prompt structure, repeated sampling, example-based prompting, and output-control parameters.

2.1 Model introduction: GPT-4

  • GPT-4 converts prompts into tokens, processes contextual relationships through transformer attention, forms intermediate representations, and decodes them into human-readable text.Temperature balances randomness and determinism, while top-k sampling restricts generation to the k most probable tokens [35].Source: §2.1 · p. 4

2.2 Giving instructions

  • Detailed instructions are essential because fundamental instructions without supplementary description tend to produce excessively general outputs [36].The examples use GPT-4, a transformer-based model fine-tuned with reinforcement learning from human feedback [4].Source: §2.2 · p. 4, §2.1 · p. 4

2.3 Be clear and precise

  • Clear, precise prompts reduce uncertainty and guide LLMs toward responses aligned with the specific requirements of a scenario.For example, specifying artificial intelligence and machine learning is more precise than asking generally about cutting-edge technology.Source: §2.3 · p. 5

2.4 Role-prompting

  • Role-prompting assigns the model a role, such as a helpful assistant, expert, or historian, to guide responses toward the desired context and output [40] [41].Triple quotes separate prompt components or encapsulate multiline strings, helping the model understand instructions in complex prompts containing quotations.Source: §2.4 · p. 5, §2.5 · p. 5

2.6 Try several times

  • Resampling runs the same prompt multiple times and selects the best response, helping overcome nondeterministic variability and increase the chance of high-quality output.The method is presented as a basic prompt-engineering practice for handling variable model responses.Source: §2.6 · p. 6

2.7 One-shot or few-shot prompting

  • One-shot prompting supplies one example, whereas few-shot prompting supplies multiple examples; few-shot prompting can add guidance for complex tasks or less capable models [44].One-shot prompting may suffice for simple tasks or highly capable models, but examples do not always help, and zero-shot prompts can sometimes outperform few-shot prompts [48].Source: §2.7 · p. 6, One-shot Prompt · p. 7

2.8 LLMs settings: temperature and top-p

  • Temperature and top-p control response randomness and diversity, but some models, including ChatGPT, expose these settings only through an API.Lower temperature produces more deterministic outputs, while top-p controls nucleus sampling.Source: §2.8 · p. 7

3 Advanced methodologies · 3.1 Chain-of-thought · 3.2 Self-consistency

Advanced prompt-engineering methods improve LLM reasoning by eliciting structured intermediate steps, aggregating diverse reasoning paths, and addressing weaknesses in complex-task accuracy. The section covers CoT variants, including zero-shot and golden CoT, alongside self-consistency and its reported applications.

3 Advanced methodologies

  • Advanced prompt-engineering techniques are introduced because foundational methods remain insufficiently accurate for complex analysis and reasoning tasks.The section aims to guide models toward more specific, accurate, and high-quality content.Source: §3 · p. 7

3.1 Chain-of-thought

  • Chain-of-Thought prompting guides LLMs through intermediate reasoning steps, significantly improving logical-reasoning accuracy and making conclusions easier for users to understand [53] [56] [57].It can use “Let’s think step by step” or manually demonstrated question-and-reasoning chains.Source: §3.1 · p. 8
  • CoT extensions broaden its scope through medical reasoning, self-education, mathematical verification, multimodal tasks, and automated rationale-chain selection [58–62].Multimodal-CoT addresses visual tasks, while Automate-CoT reduces reliance on manually crafted prompts.Source: §3.1 · p. 8

3.1.1 Zero-shot chain-of-thought

  • Zero-shot CoT enables reasoning without task examples, and appending “Let’s think step by step” produces more precise, logically coherent, and comprehensive responses [47].The approach is illustrated by comparing a standard prompt with one augmented by the step-by-step phrase.Source: §3.1.1 · p. 8, Figure 6 · p. 8, Figure 7 · p. 9

3.1.2 Golden chain-of-thought

  • Golden CoT raised GPT-4’s detective-puzzle solve rate to 83%, compared with 38% for standard CoT, but requires ground-truth reasoning solutions in the prompt.This requirement limits Golden CoT’s contribution despite its high solve rate.Source: §3.1.2 · p. 9

3.2 Self-consistency

  • Self-consistency addresses proof-planning uncertainty by generating multiple reasoning paths and marginalizing them to obtain a more accurate, convergent answer [65].The motivation arises from InstructGPT and GPT-3 difficulties selecting appropriate proof steps on PrOntoQA.Source: §3.2 · pp. 9–10
  • Self-consistency samples diverse reasoning paths and selects the answer that most consistently converges across them, improving arithmetic, commonsense, and symbolic reasoning outcomes [2, 71].The method replaces greedy decoding with sampling, then marginalizes or aggregates candidate paths; it can also combine with discriminator-guided multi-step reasoning [72].Source: §3.2 · pp. 9–10

3.3 Generated knowledge · 3.4 Least-to-most prompting

Generated knowledge improves reasoning by eliciting relevant context before answering, while least-to-most prompting decomposes complex tasks into sequential subproblems. Evidence indicates these methods enhance logical coherence, complexity generalization, and mathematical problem solving.

3.3 Generated knowledge

  • Generated knowledge prompts an LLM to produce potentially useful information before generating its final answer, especially for commonsense reasoning.The method supplies context that may be absent from the initial prompt.Source: §3.3 · p. 10
  • Generating and reusing pertinent information helps the model consider salient factors, improving the logical coherence and comprehensiveness of outputs.In the entrance example, this process prompts attention to the object’s height, which standard prompting often neglects.Source: §3.3 · p. 10

3.4 Least-to-most prompting

  • Least-to-most prompting decomposes a complex problem into simpler subproblems solved sequentially, with each solution building toward the final answer.This stepwise process enables language models to manage challenges beyond traditional prompting.Source: §3.4 · pp. 10–11
  • In the train-distance example, the method first determines speed and then uses it to calculate distance, aggregating sequential solutions into the final answer.The illustration emphasizes problem decomposition and sequential problem solving.Source: §3.4 · p. 10
  • The approach can organize detailed comparisons and identify constraints such as object height, structural integrity, size, weight, and maneuverability when reasoning about passage through an entrance.The supplied analyses contrast tanks and cars while noting that an infinitely wide entrance does not specify its height.Source: §3.4 · p. 11
  • Experiments in symbolic manipulation, compositional generalization, and mathematical reasoning found that least-to-most prompting generalizes to more complex challenges; PAL improved GSM8K [60] and SVAMP.PAL uses LLM-generated programs as intermediate reasoning steps for natural-language mathematics problems.Source: §3.4 · p. 11

3.5 Tree of thoughts · 3.6 Graph of thoughts · 3.7 Decomposed prompting

These sections present increasingly structured prompting methods: Tree of Thoughts explores alternative reasoning paths, Graph of Thoughts organizes interdependent thoughts as graphs, and DECOMP modularly assigns sub-tasks to specialized handlers. Together, they target complex reasoning through adaptive exploration, structured dependencies, and reusable decomposition.

3.5 Tree of thoughts

  • Tree of Thoughts (ToT) explores multiple reasoning paths, enabling lookahead, backtracking, and self-evaluation for more adaptive complex problem solving [77].Rather than requesting one solution, ToT generates and evaluates several candidate solutions, such as in complex mathematical problem solving.Source: §3.5 · pp. 11–12
  • ToT assesses intermediate cognitive constructs within a single prompt, providing a streamlined implementation of foundational tree-of-thought frameworks [7].A sample ToT prompt is presented in Figure 11.Source: §3.5 · p. 12, Figure 11 · p. 13

3.6 Graph of thoughts

  • Graph of Thoughts (GoT) represents LLM thoughts as vertices and their dependencies as edges, allowing arbitrary thoughts to combine into synergistic outputs [79].This graph representation supports multiple traversal methods to produce precise and comprehensive solutions for intricate, multifaceted challenges.Source: §3.6 · pp. 12–13
  • GoT is adaptable for intricate problems but requires strong subject-matter understanding and meticulous prompt design to achieve optimal outcomes [80].Its systematic graph-based reasoning can yield profound insights, while its effectiveness depends on careful construction and domain comprehension.Source: §3.6 · p. 13

3.7 Decomposed prompting

  • DECOMP [81] breaks complex tasks into simpler sub-tasks handled by specialized functions, using a decomposer-generated prompting program and an imperative controller.The program passes sub-query inputs and outputs among handlers until the final answer is obtained; handlers may be LLM prompts, symbolic functions, or learned functions.Source: §3.7 · pp. 13–14
  • DECOMP’s modularity supports independent handler optimization, error correction, hierarchical or recursive decomposition, and handler reuse across tasks.These properties facilitate systematic improvement, easier integration of new methods or models, and more efficient large-scale problem solving.Source: §3.7 · p. 15
  • DECOMP outperformed CoT in case studies involving letter concatenation, list reversal, long-context QA, open-domain multi-hop QA, and Math QA [81].Its gains came from further decomposition, recursive handling, greater example capacity, symbolic retrieval, and post-processing that corrected formatting errors; advantages over Least-to-Most Prompting can vary by task.Source: §3.7 · p. 15

3.8 Active prompt

Active prompt improves LLM reasoning by selecting and annotating the most uncertain task-specific examples rather than using traditional prefix-tuning [82]. Human-provided chain-of-thought exemplars and self-consistency guide inference while reducing engineering effort and supporting task adaptation [82].

3.8 Active prompt

  • Active prompt generates multiple predictions, estimates uncertainty through disagreement, entropy, or variance, and prioritizes informative questions for human annotation [82].The annotated answers and chain-of-thought reasoning become one-shot or few-shot exemplars for inference.Source: §3.8 · p. 15
  • Figure 14 depicts the workflow from uncertainty estimation through collection, ranking, selection, and human annotation to inference.The process illustration identifies four stages: uncertainty estimation, collection/ranking/selection, annotation, and inference.Source: Figure 14 · p. 16
  • The method enhances reasoning and task-specific prompt quality across domains while reducing human engineering effort by focusing on uncertain, informative questions [82].Self-consistency selects the most consistent answers from multiple reasoning paths, improving reliability.Source: §3.8 · pp. 15–16

3.9 Prompt pattern catalog · 3.10 Prompt optimization

Prompt pattern catalogs standardize reusable prompt designs across tasks, while prompt optimization automates prompt refinement through gradient-based, black-box, model-adaptive, planning-based, reinforcement-learning, and plugin-assisted methods.

3.9 Prompt pattern catalog

  • Prompt pattern catalogs organize 16 reusable patterns into five categories—Input Semantics, Output Customization, Error Identification, Prompt Improvement, and Interaction—for consistent, adaptable prompt design.Each pattern documents its intent, context, motivation, structure, implementation example, and practical consequences.Source: §3.9 · pp. 16–17

3.10 Prompt optimization

  • Prompt optimization systematically refines prompts to improve accuracy and relevance while reducing manual trial and error; method choice depends on task requirements, model complexity, and available resources.Gradient-based methods are directed, black-box methods broadly applicable, and model-adaptive methods customized to particular models.Source: §3.10 · pp. 17–18

3.10.1 Prompt optimization with textual gradients

  • ProTeGi generates textual gradients describing prompt flaws, applies opposite semantic updates with beam search and bandit selection, and improves performance by up to 31% over initial prompts.It works across sentiment analysis, fake news detection, and LLM jailbreak detection without access to model internals.Source: §3.10.1 · p. 18

3.10.2 Black-box prompt optimization

  • BPO improves alignment for GPT-3.5 and LLaMA-2 and outperforms RLHF and DPO alone or combined, without requiring model-internal access.It rewrites prompts using human-preference pairs, making optimization model-agnostic and interpretable.Source: §3.10.2 · p. 18

3.10.3 Model-adaptive prompt optimization

  • MAPO significantly improves question-answering, classification, and text generation over conventional task-specific optimization by adapting prompts to individual LLMs.Its two-phase process uses a warm-up dataset followed by joint SFT and RL with PPO and RRMF.Source: §3.10.3 · pp. 18–19

3.10.4 PromptAgent

  • PromptAgent uses MCTS-based strategic planning and error feedback to iteratively generate prompts, consistently outperforming human-designed and other automated prompts across general and biomedical tasks.PromptSource separately provides over 2,000 open-source prompts for roughly 170 datasets.Source: §3.10.4 · p. 19

3.10.5 Reinforcement learning

  • Reinforcement-learning prompt optimization iteratively updates prompts using reward functions that favor outputs achieving higher task-specific performance, including more accurate visual question answering.In VQA, rewards evaluate answer correctness and guide repeated prompt refinement.Source: §3.10.5 · p. 19

3.10.6 GPTs (plugins)

  • External GPT plugins directly polish prompts and reduce engineering effort, but their closed-source implementations can obscure which prompt-engineering techniques they use.Examples include Prompt Enhancer and Prompt Perfect, while Figure 15 shows GPT apps specialized in prompt generation or optimization.Source: §3.10.6 · p. 20, Figure 15 · p. 20

3.11 Retrieval augmentation

Retrieval augmentation addresses LLM hallucinations by incorporating current external knowledge into prompts, improving retrieval and generation across models and corpora. Related approaches use self-verification or unified web-based knowledge access to further support factual responses.

Problem

  • Hallucinations are fluent but unreal or inaccurate outputs caused by insufficient training evidence or overgeneralization during generation.Source: §3.11 · p. 21

Retrieval augmentation

  • Retrieval augmentation incorporates up-to-date external knowledge into prompts, including concatenated retrieved information and autoregressive retrieval-decoding for knowledge fusion.Source: §3.11 · p. 21
  • In-context retrieval-augmented models built with general-purpose retrievers significantly improve LLMs across model dimensions and diverse corpora, while RAG, FiD, and Seq2seq can reduce GPT-3 hallucinations.Source: §3.11 · p. 21
  • Chain-of-Verification has LLMs deliberate over and self-correct responses, and extending it with retrieval augmentation may provide further gains; UNIWEB treats the web as a general knowledge source.Source: §3.11 · p. 21

3.12 Reasoning and active interaction

Reasoning-and-interaction prompting expands LLM capabilities by combining multi-step reasoning with external tools and actions. ART emphasizes tool-augmented reasoning, while ReAct couples reasoning traces with task-specific actions for complex, current-data tasks.

3.12 Reasoning and active interaction

  • Together, ART and ReAct integrate reasoning with external interaction to improve the coherence, accuracy, reliability, and task range of LLM outputs.Both approaches extend prompting beyond static generation by combining logical thought processes with computational resources or actionable steps.Source: §3.12 · p. 21

3.12.1 Automatic reasoning and tool usage

  • ART combines automatic CoT reasoning with external tools, improving accuracy, relevance, and capability on tasks requiring calculations, updated information, or specialized processing [120].It breaks complex tasks into manageable steps and uses appropriate tools, supporting applications such as dynamic customer service, financial calculations, and data analysis [121, 122, 123].Source: §3.12.1 · pp. 21–22

3.12.2 ReAct framework

  • ReAct prompts LLMs to generate reasoning traces and execute actions through tools or environments, enabling efficient handling of complex, multi-step tasks [124].Its use of current external data supports more informed and reliable decisions in finance, healthcare, and legal analysis while reducing errors from outdated or incomplete information.Source: §3.12.2 · p. 22

4 Methodologies for multimodal large models

VLMs combine visual and linguistic information through transformer-based architectures, but require sophisticated prompt designs to preserve alignment, integration, and contextual coherence. The section reviews zero-/few-shot prompting and learned methods—continuous prompt vectors, CoOp, CoCoOp, and MaPLe—that improve adaptation, generalization, and multimodal task performance.

4 Methodologies for multimodal large models

  • VLMs process visual and textual information with transformer architectures, supporting image description and VQA while requiring prompt designs that maintain contextual coherence and accuracy [131].Advanced methods such as CoOp and MaPLe address data alignment, modality integration, and context preservation [131].Source: §4 · pp. 22–23

4.1 Zero-shot and few-shot prompting

  • Zero-shot prompting enables VLMs such as CLIP to classify images in previously unseen categories, whereas few-shot prompting uses a small number of inference examples to improve generalization with limited data.Prompt templates, image captions, and chain-of-thought reasoning affect zero- and few-shot VQA performance [97], while [133] adapts CLIP to few-shot classification without additional training.Source: §4.1 · pp. 23–24

4.2 Continuous prompt vectors

  • Continuous prompt vectors are learned by appending or prepending trainable random vectors, efficiently adapting frozen VLMs to video tasks such as action recognition, localization, and text-video retrieval.Only a few parameters are trained, yet the approach achieves competitive benchmark performance while lightweight Transformers model temporal dependencies.Source: §4.2 · p. 24

4.3 Context optimization

  • CoOp embeds learnable context vectors into pretrained VLMs such as CLIP and ALIGN, optimizing prompts for downstream tasks without extensive manual engineering.CoOp significantly outperforms traditional models in image recognition and VQA, while context-vector ensembling further improves robustness and generalization [138].Source: §4.3 · pp. 24–25

4.4 Conditional prompt learning

  • CoCoOp generates input-conditional prompt vectors with a lightweight neural network while keeping pretrained parameters unchanged, improving adaptation and generalization to unseen images and questions [140] [141].Dynamic prompts improve caption accuracy and robustness in VQA, image classification, and other real-world applications [142].Source: §4.4 · p. 25

4.5 Multimodal prompt learning

  • MaPLe jointly optimizes hierarchical prompts for vision and language, integrating multimodal information and significantly outperforming baseline models in image recognition and VQA [143].It addresses the limitations of optimizing vision or language prompts separately and captures dependencies across transformer layers [141] [143] [144].Source: §4.5 · pp. 25–26

5 Assessing the efficacy of prompt methods

Prompt-method efficacy is assessed through complementary subjective and objective evaluations, including human judgments, automated metrics, and standardized benchmarks. Comparisons across methods show that evaluation outcomes depend on task, model, prompting regime, and metric, with InstructEval finding limited generalization of automated prompting approaches.

5.1 Subjective and objective evaluations

  • Subjective evaluation better reflects human intuition but is costlier and slower, whereas objective evaluation is faster and cheaper, so the appropriate choice depends on the application.Automated metrics can diverge from human assessments, including the limited correlation reported for BLEU and alternative metrics.Source: §5.1 · p. 27

5.1.1 Subjective evaluations

  • Subjective evaluation uses human judgments of fluency, accuracy, novelty, relevance, and other qualities, especially for abstract outputs such as writing and summarization.Examples include human scoring of Chain of Density summaries, creative writing outputs, and moral-story responses.Source: §5.1.1 · p. 27

5.1.2 Objective evaluations

  • Objective evaluation quantitatively scores generated content with automated metrics such as BLEU, ROUGE, METEOR, and BERTScore, or tests models on benchmarks.These metrics range from reference-text similarity to higher-level semantic assessment.Source: §5.1.2 · p. 27, §5.1 · p. 27
  • Benchmark selection should match the model ability and task because no single benchmark or model performs best across all task types.Standardized datasets such as BIGbench [159] and BBH [160] support consistent and comparable assessments, while task-specific benchmarks target particular capabilities.Source: §5.1.2 · pp. 27–28
  • Objective benchmark families cover mathematical word problems, question answering, language understanding, and multimodal tasks, measuring reasoning, knowledge, text processing, and cross-modal understanding.Examples include GSM8K [60] and MATH [173], MMLU [175] and FEVER [176], SST and SummScreenFD [188], and RefCOCO [195] variants.Source: §5.1.2 · pp. 28–29

5.2 Comparing different prompt methods

  • Comparisons span prompt methods including CoT, Zero-shot, Natural Instructions (NI) [204], APO [205], APE, and ReAct, evaluated on task-specific benchmarks and metrics.Studies use benchmarks such as SVAMP, GSM8K [60], BBH [160], QuALITY, GovReport, and RefCOCO variants, but disparate models, tasks, and metrics restrict comparability [227].Source: §5.2 · pp. 29–30
  • Prompt methods are compared using task scores and additional indicators including prediction and proof accuracy, economic cost, evaluation speed, domain coverage, normalized accuracy, precision, recall, and human ratings.RAP is compared with CoT under GSM8K [217], SoT reports nearly twice the evaluation speed [219], and human-plus-automatic evaluation is used to assess alignment with human reasoning [152].Source: §5.2 · pp. 29–30
  • InstructEval [228] found that generic or omitted prompts often outperform alternatives in few-shot settings, while expert-written task-specific prompts help substantially in zero-shot settings and automated prompt generation generalizes inconsistently.Automated prompts did not outperform simple baselines in zero-shot evaluation, and performance varied across models and task types.Source: §5.2 · p. 30

6 Applications improved by prompt engineering

Prompt engineering extends LLM utility across education, creative generation, programming, reasoning, and synthetic dataset construction. Customized prompts support personalization, structured generation, debugging, multi-step reasoning, and domain-specific data creation.

6 Applications improved by prompt engineering

  • Across these applications, improved control over LLM inputs makes prompt engineering useful for real-world teaching, content creation, programming, reasoning, and other tasks.The section frames prompt engineering as a way to improve practical LLM applicability through better outputs.Source: §6 · p. 30

6.1 Assessment in teaching and learning

  • In teaching and learning, tailored prompts enable personalized content and assessment, inclusive support, and GPT-4 course-rubric generation, while automated grading and assessment analysis reduce workload and reveal learning patterns [229, 230, 231, 232, 233].Figure 18 shows GPT-4 producing a specific course rubric when given a suitable prompt.Source: §6.1 · pp. 30–31, Figure 18 · p. 30

6.2 Content creation and editing

  • For creative content, PaLM supports cross-lingual story generation, while Re3 recursively extends GPT-3 plans and DOC uses outlining, ranking, and FUDGE control to preserve long-text plot coherence [67].DOC decomposes overarching outlines into subsections, filters and ranks candidate generations, and applies a detailed controller to maintain relevance.Source: §6.2 · p. 31

6.3 Computer programming

  • In programming, self-debugging prompts combine feedback, unit tests, and explanations for text-to-SQL, MTPB decomposes implementation into multi-step prompts, and RLPG retrieves repository context for code completion [57] [238].RLPG selects a prompt through a proposal classifier and combines it with default context before generating the final output.Source: §6.3 · p. 31

6.4 Reasoning tasks

  • In reasoning tasks, few-shot reasoning traces, self-talk, chain-of-thought, customized prompts, and DIVERSE improve verbalized reasoning and reduce errors across benchmarks including MultiArith and GSM8K [60] [67] [240] [241].Zero-shot chain-of-thought substantially improves GPT-3 and PaLM over conventional zero-shot and few-shot prompting on MultiArith and GSM8K.Source: §6.4 · pp. 31–32

6.5 Dataset generation

  • Prompted in-context learning enables synthetic dataset generation for smaller domain-specific models through unlabeled annotation, training-data creation, assisted generation, and classification examples with pseudo-labels [244].The classification approach combines real dataset examples with task specifications to generate synthetic examples and pseudo-labels.Source: §6.5 · p. 32

7 LLMs security

Prompt engineering is both a mechanism for improving LLM functionality and a security framework for identifying, mitigating, and defending against vulnerabilities. The section reviews adversarial attacks, poisoning, backdoors, prompt-based attacks, model stealing, and robustness-enhancement practices.

7 LLMs security

  • Prompt engineering improves LLM performance while exposing vulnerabilities, supporting proactive security research against adversarial attacks, data poisoning, sensitive-information disclosure, and unsafe deployment [246].Continuous research into attack mechanisms and defenses is especially important for healthcare, finance, and cybersecurity applications.Source: §7 · p. 32

7.1 Adversarial attacks

  • Adversarial attacks subtly manipulate prompts or inputs, exploiting LLM sensitivity to produce incorrect, biased, harmful, or misleading outputs [249]; adversarial demonstrations can mislead models, while optimization enhances attack effectiveness, leading to incorrect legal interpretations and medical advice [254] [256] [257].Source: §7.1 · pp. 32–33

7.1.1 Data poisoning

  • Data poisoning injects malicious training data that distorts learning and causes LLMs to reproduce inaccuracies, while prompt testing and data validation can reveal or mitigate the resulting risks.Unlike trigger-based backdoors, poisoning broadly affects model performance and decision-making, with implications for healthcare, finance, and legal services [259].Source: §7.1.1 · pp. 33–34

7.1.2 Backdoor attacks

  • Backdoor attacks embed dormant vulnerabilities activated by trigger prompts, producing harmful outputs while appearing normal on clean inputs [261] [266].Prompt testing, rigorous predeployment frameworks, and transparency can help detect and mitigate attacks such as Refool, ProAttack, and Imperio [270].Source: §7.1.2 · pp. 34–35, Figure 20 · p. 34

7.1.3 Prompt injection and prompt leaking

  • Prompt injection manipulates model outputs through malicious inputs, whereas prompt leaking exposes sensitive or proprietary information embedded in prompts [252] [271].These attacks can generate harmful or misleading information and compromise application security and privacy.Source: §7.1.3 · p. 35

7.2 Prompt hacking

  • Prompt hacking strategically crafts malicious inputs to provoke unintended behavior, from benign errors to misinformation and data breaches, without requiring sophisticated technical skills [272] [273] [274] [275] [276].Defenses include prompt validation, anomaly detection, response filtering, monitoring, malicious-prompt classifiers, and developer and user training.Source: §7.2 · p. 35

7.3 Model stealing

  • Model stealing uses strategically designed query prompts to reconstruct an LLM’s mechanics, sensitive data, or surrogate behavior, particularly when the target is a black box.Consequences include intellectual-property theft, lost competitive advantage, and unauthorized cloned-model deployment [286]; defenses include query limits, anomaly detection, and defensive perturbations [292].Source: §7.3 · p. 36

7.4 Enhancing LLMs security

  • Adversarial example generation and adversarial training test vulnerabilities and improve robustness, while secure prompt design and mask filling help expose and address weaknesses [252] [293].Secure prompt engineering remains essential because unmanaged prompt engineering can introduce risks as LLMs enter critical applications.Source: §7.4 · pp. 36–37

8 Prospective methodologies

Future prompt engineering research is expected to advance through deeper understanding of AI model structures and the development of AI agents for AIGC tools. These trajectories aim to improve prompt alignment, reproducibility, explainability, and autonomous task execution.

8.1 Better understanding of structures

  • Understanding neural layers, attention mechanisms, nodes, and weights could improve prompt engineering, reproducibility, and explainable AI.Architectural understanding can reveal why prompts fail and guide their enhancement; DeepMind’s Causal Transformer illustrates efforts to understand causal relationships in data.Source: §8.1 · p. 37

8.2 Agent for AIGC tools

  • AI agents combine large models, memory, active planning, and tool use to generate solutions in complex problem spaces.Their evolution is described as models, prompt templates, chains, agents, and multi-agents, with prompt templates standardizing instructions and supporting alignment with desired outputs.Source: §8.2 · p. 38
  • Prompt engineering for AI agents must steer autonomous behavior and strategies toward self-regulation while optimizing task efficiency and precision.Agents amalgamate prompting methodologies and tools into adaptive frameworks, creating a need for prompts that guide their autonomous modulation.Source: §8.2 · p. 38

9 Conclusion

Prompt engineering is presented as essential for optimizing LLM performance and advancing AI capabilities, with methods spanning foundational prompting, advanced reasoning, multimodal learning, evaluation, and security. The review provides a framework for future research focused on model structures and AI agents.

Conclusion

  • Prompt engineering enhances LLM capabilities through clear instructions, role-prompting, chain-of-thought, and self-consistency, while CoOp and MaPLe optimize visual-textual integration in VLMs.Its efficacy is assessed through subjective and objective evaluations across education, content creation, and programming.Source: §9 · p. 38
  • Prompt engineering strengthens AI security by identifying vulnerabilities and mitigating risks through adversarial training.Source: §9 · p. 38
  • Future advances should deepen understanding of model structures and develop AI agents to further increase AI systems’ sophistication and capability.Source: §9 · p. 38

Related papers

Loading 2310.14735v6…