PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Prompting Methods for Large Language Models: A Survey

Kaiyan Chang, Songcheng Xu, Chenglong Wang, Yingfeng Luo, Xiaoqian Liu, Tong Xiao, Jingbo Zhu

arXiv:2404.01077v2cs.CL

TL;DR

Detailed LLM prompts can require substantial human effort and computational resources. This survey organizes efficient prompting into automatic prompt engineering and prompt compression, models their optimization objectives mathematically, and concludes that combining the strategies may support lightweight LLM applications.

  • Problem

    Detailed prompt design can be labor-intensive, and detailed prompt content consumes computational resources, posing barriers to practical LLM deployment.

    Source: §1 · p. 2

  • Method

    The survey reviews automatic prompt engineering and prompt compression, models their optimization objectives mathematically, and organizes compression into continuous- and discrete-space methods.

    Source: §6 · p. 22

  • Results

    The survey synthesizes two families of efficient prompting methods—automatic prompt engineering to reduce human effort and prompt compression to save computational resources—and distills their optimization objectives mathematically.

    Source: §6 · p. 22

  • Takeaways & Limitations

    The authors expect combining both resource-saving strategies to support lightweight LLM applications.

    Source: §6 · p. 22

    Automatic instructions may be task-specific, while Text-to-Vector compressed prompts are typically model-specific and challenging to transfer without fine-tuning.

    Source: §5 · p. 22

Abstract

from arXiv · show

Prompting is a mainstream paradigm for adapting large language models to specific natural language processing tasks without modifying internal parameters. Therefore, detailed supplementary knowledge needs to be integrated into external prompts, which inevitably brings extra human efforts and computational burdens for practical applications. As an effective solution to mitigate resource consumption, Efficient Prompting Methods have attracted a wide range of attention. We provide mathematical expressions at a high level to deeply discuss Automatic Prompt Engineering for different prompt components and Prompt Compression in continuous and discrete spaces. Finally, we highlight promising future directions to inspire researchers interested in this field.

1 Introduction

Prompting adapts LLMs without tuning their parameters, but detailed prompt design and content can increase human effort and computational costs. The survey organizes efficient prompting methods around reducing these resource demands, covering automatic prompt engineering and prompt compression.

1 Introduction

  • Hard prompts use discrete natural-language descriptions, while soft prompts use continuous vector representations.The survey focuses mainly on hard prompts, from concise instructions to long contexts with demonstrations.Source: §1 · p. 2
  • Manual prompt optimization is time-consuming, while detailed prompts consume computational resources, creating barriers to practical LLM deployment.The paper defines efficient prompting as achieving comparable or better performance with fewer human or computational resources.Source: §1 · p. 2
  • The authors model resource-saving methods mathematically and propose a taxonomy based on their shared optimization strategies.They also describe method pipelines and provide tables comparing methods within categories.Source: §1 · pp. 2–3
  • The survey reviews methods that reduce human effort through automatic prompt engineering and computational resources through prompt compression.Compression is divided into Text-to-Vector methods in continuous space and Text-to-Text methods in discrete space.Source: §1 · p. 3

2 Preliminary

Prompting evolved into a way to steer language models without updating their parameters, using prompts that vary in form and representation. This section distinguishes hard and soft prompts and frames efficient prompting as reducing human and computational resource use.

2.1 Origin of Prompting

  • GPT-3’s in-context demonstrations helped shift language modeling from pretrain-and-fine-tune to pretrain, prompt, and predict, steering one model across tasks without parameter updates.The section describes prompting as the current dominant interaction paradigm for language models.Source: §2.1 · p. 3

2.2.1 Hard Prompt. •

  • Hard prompts range from cloze templates and prefix examples to detailed task descriptions, with suitable input formats shaped by model architecture and pretraining objectives.The survey focuses on text-form hard prompts; soft prompts are vector representations.Source: §2.2 · p. 3, Table 1 · p. 4, §2.2.1 · p. 4

2.2.2 Soft Prompt. •

  • Soft prompts include trainable embeddings and layerwise prefixes; in the LLM era, compact vectors can store hard-prompt information with less space and accelerate inference.Earlier PLM-era methods freeze original parameters while tuning a small number of added soft-prompt parameters.Source: §2.2.2 · p. 4, Table 2 · p. 5

2.3 Mathematical Modeling

  • The survey frames efficient prompting around reducing human and computational resources, noting commonalities in their optimization objectives but limited connections between them.It models each category mathematically as a foundation for considering strategies that address both resource types.Source: §2.3 · p. 5
  • Automatic prompt engineering searches discrete hard-prompt space for instructions and demonstrations that maximize target-model performance against an evaluation metric.The model output is evaluated against ground-truth labels, using metrics such as accuracy.Source: §2.3 · p. 5
  • Prompt compression maps long text to continuous soft prompts or shorter hard prompts, filtering or summarizing while minimizing changes to the target model’s output distribution.The discrete-space approach can impose a minimum informativeness threshold.Source: §2.3 · pp. 5–6

3 Automatic Prompt Engineering

Manual prompt design for LLMs is time-consuming and difficult to optimize reliably. Automatic prompt engineering uses LLM-driven search to generate, evaluate, and select prompts, with instruction design and CoT optimization targeting different prompt components.

3 Automatic Prompt Engineering

  • Manual prompt design is time-consuming because prompts are sensitive to wording, may be suboptimal for a model, and can work in ways that conflict with human expectations.The survey identifies these challenges as sensitivity, suboptimality, and discrepancy.Source: §3 · p. 7
  • Automatic prompt engineering uses LLMs to search discrete prompt space through iterative generation, performance evaluation, and selection of candidate prompts.The approach is presented as a way to mitigate human biases and adaptively search for high-quality prompts.Source: §3 · pp. 7–8
  • Instruction design expands the instruction space, while CoT optimization constructs comprehensive problem-solving frameworks.Both use search principles and meta-prompts to guide LLMs acting as optimizers.Source: §3 · pp. 7–8

3.1 Instruction Design

The section organizes automatic prompt engineering around optimizing instructions and demonstrations, using LLMs to generate and refine candidates. It surveys sampling-, feedback-, and editing-based methods for searching prompt space.

3.1 Instruction Design

  • LLMs can generate and refine instructions from seed examples, expanding the search space that human engineers would otherwise need to construct.Meta-prompts can guide these optimization steps.Source: §3.1 · p. 8, §3.1.1 · p. 8

3.1.1 Sampling-based methods.

  • APE [126] treats instruction optimization as LLM-guided black-box search, scoring candidates generated from demonstrations and optionally resampling around the best ones.Its pipeline compresses demonstrations into an instruction and achieves human-comparable or better performance on 24/24 Instruction Induction tasks and 17/21 curated Big-Bench tasks.Source: §3.1.1 · p. 8
  • Sampling-based methods explore prompt candidates through iterative optimizer-scoring trajectories, tree search, expert clustering, or evolutionary operations.Examples include OPRO [109], PromptAgent [99], MoP [97], and evolutionary approaches such as Promptbreeder and PhaseEvo [20].Source: §3.1.1 · pp. 8–9

3.1.2 Feedback-based methods.

  • Feedback-based optimization narrows prompt search by using rewards, gradients, or preferences to indicate a clearer update direction.Feedback can come from RL, gradient-style self-feedback, or human preference.Source: §3.1.2 · pp. 9–11
  • RL-based methods use reward feedback to refine prompts, including learned prefix generation, instance-specific stimulus prompts, and hierarchical actor-critic refinement.RLPrompt [21] can produce semantically incoherent prompts; SCULPT uses preliminary structural assessment and post-execution error assessment.Source: §3.1.2 · p. 10
  • Textual-gradient methods diagnose prompt flaws and revise candidates, while related approaches aggregate feedback across samples, prompts, or data patterns.ProTeGi [72] combines textual gradients, semantic edits, Monte Carlo expansion, and beam search; other methods include prompt ensembles and multi-branched revisions.Source: §3.1.2 · pp. 10–11
  • Preference-based methods train prompt optimizers from human judgments or preference data, including paired prompts and a large-scale optimization preference dataset.FIPO [57] uses POP, with rejected GPT-3.5 outputs and chosen GPT-4 outputs reviewed by human experts and analytical models.Source: §3.1.2 · p. 11

3.1.3 Editing-based methods.

  • Editing-based methods expand discrete prompt space through lexical edits, then select candidates using search strategies or test-time reward-guided actions.GrIPS [70] uses slicing and editing with greedy, beam, or simulated-annealing search; TEMPERA [120] edits query-dependent prompts using score differences as immediate rewards.Source: §3.1.3 · pp. 11–12

3.2 CoT Optimization

The section surveys methods that optimize Chain-of-Thought prompting by generating demonstrations, refining prompts through feedback, and interacting with external tools. It also describes prompt compression across continuous and discrete spaces.

3.2 CoT Optimization

  • CoT optimization addresses the limited information in brief instructions by organizing complex problems into finer-grained subproblems and stepwise reasoning.The surveyed categories include sampling, feedback, and external-tool interaction.Source: §3.2 · p. 12

3.2.1 Sampling-based methods.

  • Sampling-based methods generate or select CoT demonstrations through self-consistency, clustering, hard-example coverage, task-specific scoring, routing, and iterative sampling.Self-consistency selects the majority answer from diverse rationale-answer samples; other methods adapt demonstration selection to task types or question clusters.Source: §3.2.1 · pp. 12–13, §3.2.2 · p. 13

3.2.2 Feedback-based methods.

  • Feedback-based methods refine reasoning prompts using prediction consistency, proxy rewards, interaction histories, or detected errors, with some methods predicting candidate scores to reduce evaluation cost.PROMST synthesizes offline error feedback, while Reprompt [12] iteratively updates common prompt components based on a loss summarized from interaction history.Source: §3.2.2 · p. 13

3.2.3 Interaction-based methods.

  • Interaction-based methods use external resources to address reasoning errors, including Wikipedia, search engines, code generation, and tool libraries.Frameworks verify or edit rationales, retrieve similar tasks, ask follow-up questions, or chain and probe tools.Source: §3.2.3 · pp. 13–14
  • Fig. 5 classifies prompt compression as Text-to-Vector methods that internalize or encode prompts and Text-to-Text methods that prune or summarize them.The caption distinguishes continuous-space and discrete-space approaches.Source: Figure 5 · p. 14

4 Prompt Compression

Prompt compression reduces the resource burden of long prompts through continuous Text-to-Vector methods and discrete Text-to-Text methods. The survey reviews internalization and encoding, then pruning and summarization, and compares pruning granularity across task types.

4 Prompt Compression

  • Long prompts can exceed context windows, contribute to forgetting when cache space is insufficient, and slow inference; compression is organized as continuous T2V or discrete T2T.T2V uses vector representations, while T2T shortens text through extractive pruning or abstractive summarization.Source: §4 · pp. 14–15, §4.2 · p. 18

4.1.1 Internalization.

  • Internalization distills prompt knowledge into model parameters, while encoding represents hard prompts with reusable soft prompts.The surveyed internalization work includes system-prompt distillation, task-specific instructions and reasoning, and retrieval-augmented generation; Distilling Step-by-Step uses rationales as additional supervision.Source: §4.1.1 · pp. 15–16, §4.1.2 · p. 16

4.1.2 Encoding.

  • Gisting encodes instructions as reusable gist tokens, while UltraGist progressively compresses context segments and accommodates sampled compression ratios for dynamic context.Gisting reports up to 26x prompt compression, 40% fewer FLOPs, and 4.2% wall-time speedups with minimal output-quality loss.Source: §4.1.2 · pp. 16–17, Figure 6 · p. 16
  • For long contexts, AutoCompressor iterates over segments, ICAE compresses the complete context at once, and SelfCP compresses only over-limit segments through a connector.The comparison figure distinguishes iterative segment compression, one-shot compression, and compression of over-limit segments; Table 3 reports that soft prompts are usually prepended for decoder-only compressors and appended for encoder compressors.Source: Figure 7 · p. 17, §4.1.2 · pp. 17–18, Table 3 · p. 18

4.2.1 Pruning.

  • Pruning removes less-informative units without changing the original prompt’s linguistic expression, using coarse-grained, coarse-to-fine, or fine-grained schemes.Coarse-grained methods select demonstrations, sentences, or documents; LLMLingua allocates compression budgets across components and filters tokens, while LongLLMLingua adds question-aware document and token selection.Source: §4.2.1 · pp. 18–19
  • Selective Context finds phrase-level filtering most effective, reducing inference memory usage by 36% and inference time by 32% with negligible performance drop.Other fine-grained approaches use relation-aware graphs or learned token inclusion decisions; LLMLingua-2 reports 3x-6x compression speedup and 1.6x-2.9x end-to-end latency acceleration.Source: §4.2.1 · pp. 19–20
  • Coarse-to-fine pruning appears more beneficial for complex reasoning, while fine-grained pruning is more suitable for long-context tasks.Table 4 cautions that differing experimental setups can make method performances not directly comparable.Source: §4.2.1 · p. 20, Table 4 · p. 20

4.2.2 Summarization.

  • Summarization compresses prompts semantically, either by training on original outputs or by checking in real time whether a summary retains enough information to answer.Examples include query-focused RECOMP summaries for retrieved documents, MEMWALKER’s memory-tree navigation, and CompAct’s iterative summarization, which reports 47x compression.Source: §4.2.2 · pp. 20–21

5 Future Directions

The survey identifies opportunities to combine automatic prompt engineering with compression and to improve efficient prompting’s robustness and alignment with human understanding. It also notes continued reliance on human input and model capabilities.

Combining strategies and human interpretability

  • Future work could combine iterative CoT improvement with compression, or optimize both synchronously using performance rewards and balancing factors for quality, cost, and speed.The survey also proposes investigating whether human interpretability is an appropriate standard for information useful to LLMs.Source: §5 · p. 21

Robustness and open questions

  • Prompt transfer across data distributions and models remains an open direction; automatic prompt engineering still requires human-supplied examples or initial prompts and careful meta-prompt design.The survey also says optimization relies heavily on LLMs given SLM reasoning limits, so model capability and computational cost need consideration.Source: §5 · p. 22

6 Conclusion

The survey reviews automatic prompt engineering to reduce human effort and prompt compression to reduce computational resources, modeling category objectives mathematically and highlighting future directions.

6 Conclusion

  • The survey synthesizes representative efficient prompting methods with an emphasis on efficiency and identifies potential future directions and open-source projects.It presents combining resource-saving strategies as a future prospect for lightweight LLM applications.Source: §6 · p. 22

A.1 Open Resources

The appendices list open resources for prompt compression and automatic prompt engineering.

A.1 Open Resources

Loading 2404.01077v2…