Source-linked AI summary
Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4
Sondos Mahmoud Bsharat, Aidar Myrzakhan, Zhiqiang Shen
TL;DR
Reliably guiding pretrained language models toward specific responses remains challenging. The paper presents 26 prompt-design principles and evaluates them across models; its reported GPT-4 results show higher response quality and accuracy with specialized prompts.
Problem
Reliably guiding pretrained language models to produce specific responses and fully use their capabilities remains a challenge for users designing prompts.
Source: §1 · pp. 1–2Method
The paper presents 26 prompt-design principles and evaluates their effects on model responses using human assessment of principled and original prompts.
Source: §3.3 · p. 7, §4.2 · p. 8Results
57.7% and 36.4% average improvements in GPT-4 response quality and accuracy, respectively, are reported for specialized prompts on ATLAS.
Source: §1 · p. 3Takeaways & Limitations
The authors conclude that principled prompting can improve response relevance, brevity, and objectivity within their experiments.
Source: §5 · p. 11The evaluation covers seven models and a limited selection of questions, and untested architectures may respond differently to the principles.
Source: §6 · p. 19
Abstract
from arXiv · showhide
This paper introduces 26 guiding principles designed to streamline the process of querying and prompting large language models. Our goal is to simplify the underlying concepts of formulating questions for various scales of large language models, examining their abilities, and enhancing user comprehension on the behaviors of different scales of large language models when feeding into different prompts. Extensive experiments are conducted on LLaMA-1/2 (7B, 13B and 70B), GPT-3.5/4 to verify the effectiveness of the proposed principles on instructions and prompts design. We hope that this work can provide a better guide for researchers working on the prompting of large language models. Project page is available at https://github.com/VILA-Lab/ATLAS.
1 Introduction
Reliable prompting remains difficult, and the paper presents 26 principles for improving prompt design and eliciting responses that better match users’ intentions. Its experiments report higher GPT-4 response quality and accuracy with specialized prompts.
1 Introduction
- Prompt engineering is presented as an alternative to task-specific fine-tuning, which the authors describe as impractical or unattainable for many users and developers.The supplied passages define prompt engineering as crafting precise, task-specific natural-language instructions and selecting representative examples.Source: §1 · pp. 1–2
- Figure 1 contrasts original GPT-4 prompts and responses with principled versions using principles 5 and 6.The caption identifies the left side as original and the right side as principled prompts and their associated responses.Source: Figure 1 · p. 2
- The paper introduces comprehensive principles for improving prompts and reports that more precise directives and audience cues can help elicit responses aligned with expectations.It also reports that larger models show considerable capacity for simulation and that assigning a specific role can improve alignment with intended results.Source: §1 · p. 2
- 57.7% and 36.4% average improvements in GPT-4 response quality and accuracy, respectively, are reported for the specialized prompts.The evaluation uses the manually designed ATLAS benchmark.Source: §1 · p. 3
2 Related Work
The related work situates the study within the development of large language models and the growing field of prompt design. It reviews approaches ranging from intermediate reasoning steps to generated auxiliary prompts.
Large Language Models
- The review traces language-model development from BERT, T5, and early GPT models through larger systems including GPT-3, GPT-4, and Gemini.It also discusses Gopher, LLaMA, Chinchilla, and Mistral in relation to scale, efficiency, and capability.Source: §2 · p. 3
Prompting
- Prompting research is described as studying how prompt design affects model performance, including few-shot and zero-shot settings and sensitivity to semantic and contextual changes.The review notes that prompting can be used without fine-tuning.Source: §2 · p. 3
- Reviewed strategies include aggregating imperfect prompts, generating intermediate reasoning steps, decomposing tasks, using explanations, and generating auxiliary prompts.The examples include Ask-Me-Anything, Chain-of-Thought, least-to-most prompting, and Directional Stimulus Prompting [12].Source: §2 · p. 4
3 Principles
The principles are motivated by the need to tailor prompts to model behavior and task context. The paper groups 26 principles into five categories and describes guidance for clarity, context, task alignment, examples, bias, iteration, and complex prompts.
3.1 Motivation
- The authors motivate prompt customization as a way to improve output quality, arguing that effective prompting requires understanding model behavior and response principles.They present 26 principles for different scenarios and circumstances.Source: §3.1 · p. 4
3.2 Overview
- The 26 principles are grouped into five categories: prompt structure and clarity, specificity and information, user interaction and engagement, content and language style, and complex tasks and coding prompts.Examples include specifying the intended audience, requesting unbiased answers, inviting clarifying questions, writing directly, and breaking complex tasks into simpler prompts.Source: §3.2 · p. 4, Table 1 · p. 5, Table 2 · p. 6
3.3 Design Principles
- Design guidance includes concise, clear, context-relevant, task-aligned prompts, examples for complex tasks, bias awareness, incremental prompting, and iterative refinement based on model responses and feedback.Advanced prompts may also use conditional statements, logical operators, or pseudocode to guide reasoning.Source: §3.3 · p. 7
4 Experiments
Experiments on ATLAS compare principled prompts with unmodified prompts across model scales, evaluating response-quality boosting and correctness. The results report improvements across scales, with individual-model findings and examples further illustrating the outcomes.
4.1 Setup and Implementation Details
- The evaluation uses ATLAS, a manually crafted benchmark with standard and challenging subsets, and compares paired responses with and without principles using human evaluation.For each principle and the challenging subset, the benchmark includes 20 human-selected questions; each question receives a single response.Source: §4.1 · pp. 7–8
4.2 Models and Metrics
- The study evaluates instruction-finetuned LLaMA-1/2 models at 7B and 13B, LLaMA-2-70B-chat, GPT-3.5, and GPT-4 using separate boosting and correctness settings.Boosting uses simpler tasks to assess response-quality improvement, while correctness uses complex reasoning tasks; principles 14, 15, 21, 22, and 23 are not applicable in the correctness setting.Source: §4.2 · pp. 8–9
4.3.2 Results on individual LLMs
- Individual-model results show a stable average 50% response-quality improvement, and correctness gains increase with model size from LLaMA-2-13B through LLaMA-2-70B-chat, GPT-3.5, and GPT-4.Figures 10–12 provide per-principle boosting and absolute and relative correctness results for individual models.Source: §4.3.2 · pp. 10–11
4.3.1 Results on small, medium and large-scale LLMs
- Correctness averages 20%–40%; absolute accuracy is 10%–40% for small and medium models and above 40% for large models, while relative improvement exceeds 10% overall and can exceed 20% for larger models.Boosting improves response quality across all three scales; principles 2, 5, 15, 16, 25, and 26 show the greatest improvement for large-scale models, and principle 14 improves all questions to which it is applied.Source: §4.3.1 · pp. 9–10, Figure 5 · p. 10, Figure 4 · p. 10
4.3.3 More examples on various scales of LLMs
- Additional examples for LLaMA-2-7B and LLaMA-2-13B report that the principles enhance response accuracy.The examples are shown in Figures 13–16.Source: §4.3.3 · p. 11
5 Conclusion
The paper presents 26 principles for focusing LLMs on crucial input context and reports improvements in response quality. Its experiments use constrained-shot prompting, while future work could explore other ways to refine or integrate the principles.
5 Conclusion
- The 26 principles help LLMs focus on crucial input context, with reported gains in response relevance, brevity, and objectivity.The paper says the principles can reformulate contexts that might otherwise compromise output quality.Source: §5 · p. 11
- The principles are provided before input processing to encourage models to produce better responses.Source: §5 · p. 11
- The experiments used constrained-shot prompting, and future work could test fine-tuning, reinforcement learning, direct preference optimization, or alternative prompting methods.Source: §5 · p. 11
- Successful strategies could be integrated into standard LLM operations by training on principled prompts and polished principled responses.Source: §5 · p. 11
6 Limitations and Discussion
The principles may be less effective on very complex or highly specialized questions, and results are bounded by the models, question sample, and human assessment criteria used.
6 Limitations and Discussion
- Effectiveness may diminish on very complex or highly specialized questions, depending partly on each model’s reasoning capabilities and training.Source: §6 · p. 19
- Models with architectures unlike the seven tested may respond differently to the principles.Source: §6 · p. 19
- Improvement and correctness percentages reflect a limited question selection, and assessment criteria and results may vary across evaluators.The authors suggest expanding the question set to support more generalized findings.Source: §6 · p. 19