Source-linked AI summary
Beyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on Quality
Qipeng Xie, Zi Liang, Jiafei Wu, Yufei Chen, Weizheng Wang, Wenao Ma, Zhong Ming, Haiqin Yang, Kaishun Wu
TL;DR
LLMs are highly sensitive to minor prompt wording changes, while existing approaches provide limited fine-grained mechanistic understanding of prompt robustness. The paper analyzes 132,000 prompt variants at n-gram level, identifies linguistic anchors and a performance–stability scaling law, and evaluates an automated refinement agent. The agent reduces performance variance by 40.7% in code generation while maintaining or improving mean performance.
Problem
Prompt instability makes minor lexical variations produce substantial performance fluctuations, while existing analyses offer limited fine-grained understanding of the mechanisms governing robustness.
Method
The study performs n-gram–level statistical analysis over 132,000 prompt variants and develops an automated Prompt-Refining Agent based on domain-specific terminology and explicit action directives.
Results
40.7% variance reduction in code generation is achieved while maintaining or improving mean performance, alongside a scaling law linking higher average performance with greater stability.
Takeaways & Limitations
Robustness is associated with precisely specifying the model’s interpretation space through domain anchoring and explicit operational constraints.
Takeaways & Limitations
The findings are primarily based on Qwen and Gemini models and English WizardLM prompts covering code, math, and creative writing, so broader generalizability remains unverified.
Abstract
from arXiv · showhide
Large Language Models (LLMs) exhibit extreme sensitivity to surface-level prompt variations, in which minor lexical changes can trigger disproportionate performance fluctuations. Moving beyond black-box optimization and coarse-grained templates, we present the first large-scale, n-gram token-level mechanistic analysis of prompt stability, leveraging a dataset of 132,000 prompt variants. Our investigation reveals a fundamental Scaling Law of Prompt Performance Stability: higher average task performance is strongly associated with lower variance and greater robustness across prompt perturbation. We identify two core linguistic drivers underlying this robustness: (1) Domain-Specific Terminology, which tightly anchors semantic boundaries, and (2) Explicit Action Directives, which formalize reasoning trajectories. Together, these elements constrain the model's interpretative space, effectively ``locking in'' more deterministic generation behavior. Building on these insights, we introduce an automated Prompt-Refining Agent that systematically restructures input queries by injecting domain anchoring and operational constraints. Empirical evaluation shows that our approach reduces performance variance by 40.7% in code generation task, while preserving or improving mean performance. These findings provide a statistically grounded and mechanistically interpretable framework for achieving robust prompt engineering.
1 Introduction
Prompt instability makes minor lexical changes produce substantial performance fluctuations, while its mechanisms remain poorly understood. This study analyzes prompt sensitivity at n-gram scale and identifies robustness patterns and an automated refinement approach.
- Prompt instability causes substantial performance fluctuations from minor lexical variations, undermining reliability and reproducibility in LLM deployment.
- The study analyzes 132,000 prompt variations across 12,000 instructions and 11 stylistic variants to characterize token-level robustness patterns.
- Higher average task performance consistently corresponds to greater stability under stylistic perturbations.
- Domain-specific terminology and explicit action directives underpin high-robustness prompts by constraining interpretation and reasoning trajectories.
- The automated Prompt-Refining Agent reduces performance variance by 40.7% while maintaining or improving mean performance.
2 Related Work
Prior work mainly optimizes performance or treats prompt sensitivity as noise through calibration and consistency methods. This paper instead frames sensitivity as a structured statistical phenomenon and seeks its linguistic mechanisms.
- Existing methods use inference-time calibration and training-time consistency constraints to mitigate instability across prompts.
- Unlike approaches that eliminate sensitivity as noise, this work treats it as a structured statistical phenomenon.
- The study aims to uncover intrinsic linguistic properties of high-performing prompts that explain their robustness.
3 Methodology
The methodology stress-tests prompts with five stylistically distinct rewriting strategies and evaluates whether they probe independent perturbation dimensions. Embedding and lexical analyses characterize their separation, coverage, and intensity.
- 3.1 Systematic Perturbation Strategies: Five semantically equivalent rewriting strategies simulate diverse user inputs across distinct stylistic dimensions.
- 3.2.1 Orthogonality of Perturbation Directions: Each rewritten instruction is evaluated through an embedding delta vector computed relative to the original instruction.
- 3.2.1 Orthogonality of Perturbation Directions: PCA shows distinct perturbation clusters, with Syntactic Noise and Added Contextual Information farthest from the origin while Instructional Augmentation remains near it.
- 3.2.1 Orthogonality of Perturbation Directions: The five strategies have approximately 0.22 average off-diagonal cosine similarity, with Syntactic Noise near 0.12 against the others.
- 3.2.2 Lexical and Informational Coverage: Syntactic Noise covers surface-form changes, whereas Added Contextual Information produces systematic information expansion.
- 3.2.3 Perturbation Intensity: Average semantic distances range from approximately 0.07 for Instructional Augmentation to 0.36 for Syntactic Noise.
4 Experiments
The experiments use a large-scale perturbation benchmark and automated evaluation pipeline to quantify prompt performance and sensitivity. They identify a negative relationship between mean performance and response variability, with robustness explaining substantial performance variance.
- Experimental setup: 12,000 instructions generate 132,000 prompt evaluations through one original and ten stylistic variants per instruction.The benchmark uses the WizardLM_evol_instruct_70k dataset and evaluates responses with a three-model rewriting, generation, and judging pipeline.
- Evaluation metrics: The evaluation computes Mean Score (µ) and Standard Deviation (σ) across each group of 11 samples to quantify performance and sensitivity.Quality is assessed with a continuous 1-100 composite score using a utility hierarchy that prioritizes Accuracy, Completeness, Clarity, and Practical Value.
- Scaling law: Figure 4 shows a negative correlation between mean performance and response variability, modeled using Ordinary Least Squares regression.The regression relates mean performance to prompt-performance standard deviation.
- Scaling law: −1.083 slope and R2 = 0.70 indicate that greater performance variability is associated with lower mean performance and explains 70% of observed performance variance.A one-unit increase in standard deviation corresponds to an average decrease of approximately 1.08 points in mean performance.
- Implications: The scaling relationship motivates reporting prompt sensitivity explicitly and prioritizing high-performance, low-variability prompt designs for deployment.The paper frames robustness as a fundamental dimension of model capability rather than transient measurement noise.
5 Token-Level Analysis
A token-level analysis identifies linguistic patterns associated with prompt robustness and evaluates an automated agent that injects those patterns into prompts. Across six tasks, the intervention generally reduced variability while often preserving or improving performance, with task-dependent trade-offs in open-ended domains.
- Analysis and stratification: The HP-HR quadrant isolates prompts combining high performance with high robustness using median mean-score and standard-deviation thresholds.Distinctive n-grams are then extracted with Log Odds Ratio and add-k smoothing.
- Key linguistic patterns: Domain-specific terminology narrows the semantic search space, while explicit action directives constrain response structure and execution paths.Together, these patterns reduce interpretation space by specifying both the semantic domain and the structural template.
- Pattern injection: The Prompt-Refining Agent injects domain anchoring and constraint refinement to transform generic queries into more robust instructions.Examples include replacing “fix the code” with a domain-anchored debugging instruction and adding step-by-step or bullet-point directives.
- Quantitative evaluation: 40.7% lower Rouge-L Std and 40.4% lower LLM-as-a-Judge Std were achieved for Code Writing after pattern injection.The LLM-as-a-Judge Mean Score also increased by 7.9%, while Std decreased from 12.96 to 7.73.
- Quantitative evaluation: Creative Writing and Health Medical traded lower Mean Scores for substantially lower variance under added structural constraints.Variance reductions reached 46.9% and 53.1% for Creative Writing, and 50.1% and 50.2% for Health Medical across the reported metrics.
- Case study: The case study shows that domain terms and explicit directives narrow ambiguous code-generation paths toward standardized outputs.The improved prompt replaces unconstrained sorting instructions with constraints such as “QuickSort” and explicit directives.
6 Conclusion
The study finds that high-performing prompts are associated with lower output variance and greater robustness. Its Prompt-Refining Agent reduces performance variance by up to 40.7% in code generation tasks.
- High-performing prompts are intrinsically associated with low output variance under stylistic perturbations.
- The Prompt-Refining Agent reduces performance variance by up to 40.7% in code generation tasks.
Limitations
The findings are bounded by the evaluated models, language, benchmark, task types, and automated evaluation procedure. Their generalization to other architectures, languages, reasoning tasks, and human judgments remains to be verified.
- Experiments primarily used Qwen and Gemini models, so scaling laws and token patterns may differ in smaller or Mixture-of-Experts architectures.
- The analysis covered English prompts from WizardLM tasks in code, math, and creative writing, leaving low-resource languages and highly abstract reasoning tasks unverified.
- Automated judges may have preferences that differ from human evaluation, especially in subjective domains such as creative writing.
A.1 Rewrite Prompt
The rewrite prompt constructs the Sensitivity Manifold by generating semantically preserved prompt variants across five orthogonal rewriting strategies.
- The Sensitivity Manifold contains 132,000 prompt variants generated by Gemini-2.5-flash.
- The rewriting engine preserves core task semantics and any few-shot examples while varying prompts across five orthogonal strategies.
A.2 Evaluation Prompt
The evaluation prompt uses Grok-4-fast to score automated quality continuously across weighted dimensions, reducing clustering around high scores.
- Grok-4-fast evaluates prompt quality with a continuous 1-100 scoring mechanism and weighted dimensions.
- The scoring design targets the clustering effect in LLM judges, where scores often concentrate around 7-10 on a 10-point scale.
A.3 Qualitative Comparison of Prompt Variants
The qualitative comparisons contrast fragile original prompts with robust variants generated through systematic rewriting strategies while preserving the underlying task. Across examples, robust prompts add domain anchoring, explicit constraints, structural guidance, or narrowed scope to shape responses.
- Rewriting setup: The Prompt-Refining Agent generates robust variants from original instructions using five rewriting strategies while preserving the underlying task.The strategies include semantic restructuring, irrelevant context, style shifting, vagueness and conditional framing, and structural noise; examples must also be preserved, although their position may change.
- Evaluation setup: The evaluation prompt scores candidate answers on a continuous 1-100 scale to encourage full-range utilization and fine-grained differentiation.The prompt requests valid JSON output for multiple candidate answers.
- Logic Puzzles: In logic puzzles, the robust prompt anchors the domain to “algorithmic sequence computation” and supplies a structural explanation template.The comparison uses a traveler-and-villagers hand-gesture puzzle and includes a Ruby solution context.
- Domain and procedural constraints: In health, format conversion, and mathematics, robust prompts respectively enforce safety boundaries, ground abstract requests in engineering simulations, and specify exact implementation procedures.The corresponding examples involve anxiety-related health data, PHP parsing of JSON objects, and a mountain-peak puzzle with restrictions on built-in sorting and maximum functions.
- Scope control: In creative writing and code writing, robust prompts isolate generation scope and prioritize engineering robustness over open-ended feature expansion.The examples cover generating a Markdown-formatted plot twist in C# and improving a Ruby web-scraping script.