Source-linked AI summary
Predicting LLM Performance from Prompt Linguistic Features: An Empirical Study in Requirements Engineering
Quim Motger, Alessio Miaschi, Xavier Franch, Mohammad Amin Zadenoori, Alessio Ferrari
TL;DR
LLM performance varies with prompt formulation, but practitioners lack a low-cost way to assess linguistic effects before inference. The study profiles 9,000 controlled prompt variants and models their performance across five open-source LLMs. Linguistic features significantly predict all evaluated targets, with structural features providing the main signal and supporting interpretable pre-inference prompt screening.
Problem
Existing prompt-engineering approaches do not explain which linguistic properties matter, while automatic optimisation requires repeated, costly LLM evaluations.
Method
The study combines linguistic profiling, controlled prompt variation, and regression-based analysis of binary requirements classification across multiple LLMs.
Results
Linguistic features significantly predict prompt performance across all targets, with predictive signal concentrated in syntactic and morphosyntactic structure.
Takeaways & Limitations
Linguistic profiling can provide a fast, human-interpretable pre-filter for candidate prompts before costly LLM optimisation.
Takeaways & Limitations
The study is scoped to one task, dataset, and prompt component, so its findings are validated within a controlled context rather than across broader settings.
Abstract
from arXiv · showhide
Background. LLM outputs are highly sensitive to prompt formulation: small wording changes can substantially affect output quality. This matters in software engineering, where prompts guide requirements analysis, code generation, and artefact synthesis. Poor formulations yield unreliable artefacts, yet practitioners lack principled ways to assess a prompt before inference, making selection depend on costly LLM calls and trial-and-error refinement. Aims. We investigate whether measurable linguistic properties of prompts can predict LLM performance before inference, enabling low-cost prompt selection and refinement, validated on binary requirements classification targeting F1, F2, precision, and recall. Method. We generate 9,000 linguistically controlled prompt variants from 100 initial prompts by varying 30 linguistic metrics, evaluated with five open-source LLMs on 625 annotated requirements. Regression predictors are trained via stratified 10-fold cross-validation with permutation-based significance testing; feature importance analysis identifies cross-LLM and model-specific predictors. Results. Linguistic features significantly predict prompt performance across all targets (R2 in [0.38,0.42], q<0.05). Syntactic and morphosyntactic features drive most predictive signal; cross-LLM predictors include compound dependency distribution, conjunction density, and word/sentence length, reflecting sensitivity to domain vocabulary and complex structures. Conclusions. Results suggest practical implications for prompt engineering, including overlap between linguistic patterns that reduce LLM performance and those that increase human comprehension difficulty, and the irrelevance of lexical variety as a quality dimension. More broadly, linguistic profiling combined with standard regression provides an effective, interpretable, low-cost prior before costly optimisation pipelines.
1 Introduction
LLM performance is highly sensitive to prompt formulation, while existing structural approaches and automatic optimisation do not explain or cheaply assess the effects of linguistic variation. This study therefore tests whether measurable prompt properties predict performance in binary requirements classification.
- Small changes in prompt wording, structure, or phrasing can substantially alter LLM output quality, including binary requirements judgements.
- Structural and template-based prompt engineering fixes designs, whereas automatic prompt engineering evaluates rewritten candidates through repeated LLM calls.The latter is computationally expensive and energy-intensive.
- Measurable linguistic properties are hypothesised to predict LLM performance before costly optimisation pipelines are used.The intended benefit is reducing the number of LLM interactions during prompt development.
- The study validates this hypothesis in binary requirements classification, measuring performance with controlled linguistic manipulation and ISO/IEC 25010 functional correctness.
- The contributions include controlled paraphrasing, 9,000 prompt variants evaluated on five open-source LLMs, and ranked cross-LLM and model-specific linguistic predictors.
2 Background
Prompt engineering formulates and refines textual inputs to maximise task performance, while linguistic profiling represents texts as interpretable quantitative feature vectors. These representations support downstream analyses without LLM inference and are applied here to prompt-performance modelling.
- 2.1 Prompt Engineering: Prompt engineering designs and refines textual inputs to maximise an LLM’s performance on a specific task.It can be performed manually or automatically through candidate generation and evaluation.
- 2.1 Prompt Engineering: In requirements classification, inputs are textual requirements, outputs are manually annotated classes, and accuracy or F1 can evaluate predictions.
- 2.2 Linguistic Profiling: Linguistic profiling characterises text through measurable features extracted from automatic linguistic annotation.The approach has been used in stylometry, readability assessment, language identification, and neural language-model evaluation.
- 2.2 Linguistic Profiling: Profiles represent texts as quantitative descriptors spanning raw text, lexical, morphosyntactic, and syntactic levels.Examples include length, type/token ratio, part-of-speech distributions, dependency relations, and parse-tree depth.
- 2.2 Linguistic Profiling: Each feature function maps an automatically annotated text to a real-valued descriptor, forming a vector usable for classification, regression, correlation, or distributional analysis.The study uses standardised formalisms such as Universal Dependencies for linguistic annotation.
- 2.2 Linguistic Profiling: Linguistic profiling is interpretable, deterministic, inexpensive, and reproducible because its features are computed from raw text without learned parameters or LLM inference.Prompt variants can therefore be represented as structured feature vectors for regression-based analysis.
3 Research Method
The study combines a large-sample analysis of controlled prompt variants with a laboratory experiment across multiple LLMs. Cross-validated regression tests predictive feasibility, while permutation importance identifies relevant linguistic features and their stability across models.
- 3 Research Method: The research design combines a sample study of prompt variants with a controlled laboratory experiment using multiple LLMs.
- 3 Research Method: The study asks whether linguistic features predict binary requirements-classification performance and which features have the highest predictive relevance.
- 3 Research Method: Stratified 10-fold cross-validation evaluates prediction across Precision, Recall, F1, and F2 using five regression models.
- 3 Research Method: Permutation-based feature importance is applied to the best-performing regression setup to compare individual and grouped predictors across LLMs and performance targets.
- 3 Research Method: Joint regression with cross-validation and permutation-based significance testing evaluates reliable signal in a high-dimensional, correlated feature space.Feature-importance analysis then recovers interpretable contributions.
- 3 Research Method: Stage 1 generates and validates linguistically controlled prompt variants; Stage 2 evaluates them on binary requirements classification and models the resulting prompt-performance pairs.
3.2 Stage 1: Experimental Space Construction
Stage 1 constructs a controlled experimental space for binary requirements classification by paraphrasing fixed few-shot prompts along targeted linguistic dimensions. The study uses 625 annotated requirements, 100 initial prompt variants, and Profiling-UD features to validate and characterise the generated space.
- Task and Dataset: The task classifies software requirements as Functional or Non-functional using the PROMISE-NFR dataset.The dataset contains 625 annotated requirements.
- Prompt Construction: The study starts from 100 few-shot prompt variants whose class definitions differ in wording while preserving the same underlying concepts.The overall prompt structure remains constant across variants, while linguistic manipulation targets only the class definitions.
- Experimental Scope: The experimental design fixes the task, dataset, and few-shot strategy to enable an in-depth analysis of linguistic variability within one controlled setting.This deliberately trades breadth across tasks and prompting strategies for experimental control.
- Linguistic Manipulation: Controlled paraphrasing targets 30 metrics across raw text, parse-tree, verbal-predicate, and subordination groups.The full Profiling-UD feature set, covering nine groups, is computed for predictive analysis.
- Variant Generation: A paraphrasing template instructs Qwen3:32B to modify a specified linguistic property while preserving semantic meaning.The template specifies the metric, its description, target range, and the paired functional and non-functional definitions.
3.3 Stage 2: Performance Evaluation and Prediction
Stage 2 evaluates generated prompts on five open-source instruction-tuned LLMs and predicts their classification performance from linguistic features. Regression models are assessed across four targets using stratified cross-validation, permutation significance testing, and feature-importance analysis.
- Performance Evaluation: The 9,000 prompt variants are evaluated with five open-source instruction-tuned LLMs on F1, F2, precision, and recall.The model suite includes Qwen2, Falcon3, Granite, Ministral, and Meta-Llama instruction-tuned models.
- Prediction Setup: Each prompt is represented by a 124-dimensional Profiling-UD feature vector, with targets aggregated over 625 requirements per variant.The regression targets are F1, F2, precision, and recall.
- Regression Models: Five regression models span interpretable linear and tree-based methods through nonlinear ensembles and a multilayer perceptron.The models are Decision Tree, Linear SVM, Random Forest, Gradient Boosting, and MLP.
- Validation: Model performance is evaluated with stratified 10-fold cross-validation, with hyperparameters tuned by randomized search using R2.Stratification is based on experimental condition.
- Inference and Interpretation: Statistical significance is tested by 500 R2 permutations with Benjamini-Hochberg false-discovery-rate correction.Feature-family and feature-level analyses then examine predictive contributions and cross-LLM stability.
4 Results
The results support predicting prompt performance from linguistic features, while showing that controllability is partial and structurally constrained. Predictive signal is concentrated in syntactic and morphosyntactic properties, with broadly consistent but partly model-specific feature rankings.
- Linguistic Variability: 61.5% of targeted metrics moved in the expected direction under maximise, compared with 46.2% under minimise; 30.8% were bidirectionally controllable.The equal condition was closer to baseline than both extremes for 61.5% of metrics.
- Linguistic Variability: The controlled variants increased dispersion, with a median IQR variability gain of approximately 2.12× and increases for 88.9% of metrics with non-zero baseline IQR.This broadened linguistic coverage and provided a diverse experimental space.
- Predictive Feasibility: RF significantly predicts all four performance targets, with R2 in [0.38, 0.42] and q < 0.05.RF achieves an average R2 = 0.405 across the target metrics.
- Feature Groups: Syntactic Relations and Morphosyntactic Information provide the strongest single-group signal, with R2 = 0.375 and R2 = 0.331 versus a full-feature baseline of R2 = 0.418 for Avg_F1.Parse Tree Structure contributes moderately at R2 = 0.210, while remaining groups are near zero individually.
- Feature Importance: The top predictors are compound dependency distribution, character-per-token ratio, and conjunction density, reflecting technically dense and enumerative requirements language.The top-ranked compound feature has mean rank 1.0 and importance 0.21–0.33.
- Cross-LLM Stability: Feature rankings are highly consistent across LLMs, with pairwise Spearman ρ ranging from 0.86 to 0.95, although some features remain model-specific.Rank consistency across performance targets reaches ρ ≥0.97, while model selection remains a relevant design variable.
- Feature Importance: Lexical diversity is consistently uninformative, with TTR features having mean rank > 95 across models and targets.Vocabulary variability does not predict prompt performance in this study.
5 Discussion
Prompt performance depends not only on semantic content but also on its linguistic packaging, with predictive signal concentrated in structural features. These findings motivate linguistic profiling as an interpretable pre-inference scoring signal and support targeted prompt refinement.
- Interpretation: Linguistic packaging contributes predictive signal beyond prompt semantics, with performance linked to specific structural properties.The discussion identifies morphosyntactic patterns, long dependency arcs, and dense coordination as concentrated sources of signal.
- Interpretation: Linguistic features associated with lower prompt performance overlap with structures linked to increased human comprehension cost.The discussion connects dependency-length effects and nominalisation patterns with the identified prompt-performance features.
- Practical role: Linguistic fragility can be characterised before inference through measurable syntactic and morphosyntactic properties rather than only through post-hoc diagnostics.This reframes linguistic profiling as a pre-inference scoring signal for prompt engineering.
- Interpretation: Raw-text measures such as length and sentence count act as proxies for deeper structural density rather than explanatory factors themselves.Associated phenomena include compound noun phrases, long dependency arcs, coordinated nominal lists, and dense technical terminology.
- Practical implications: Lexical variety is irrelevant to prompt effectiveness, so prompt refinement should target structural density rather than vocabulary richness.The discussion recommends preserving necessary terminology while reducing unnecessary syntactic and nominal density.
6 Threats to validity
The study’s validity is constrained by its measurement choices, prompt-generation model, and controlled scope, although consistent findings across diverse models support the conclusions.
- Internal validity: Prompt variants are generated exclusively with Qwen3:32B, leaving effects of alternative paraphrasing models for future assessment.Generation is treated as instrumental, and metric expansion is empirically validated for the study’s purposes.
- Construct validity: Permutation importance may distribute predictive contribution arbitrarily among correlated linguistic features.Group-level contributions are reported alongside individual rankings to partially mitigate this issue.
- Construct validity: The four performance indicators derive from binary requirements classification and may not generalise to prompt quality in other contexts.The study also considers possible PROMISE-NFR pre-training leakage; low mean Jaccard scores of [0.105, 0.165] provide no strong evidence of systematic reconstruction.
- External validity: The study covers one task, dataset, and prompt component, despite analysing 9,000 variants, 124 metrics, five LLMs, and five regression models.This design prioritises experimental control; generalisation to other tasks, datasets, and prompting strategies remains open.
- Conclusion validity: Consistent patterns across architecturally diverse LLMs suggest predictors reflect task and prompt structure rather than model-specific idiosyncrasies.Newer models may nevertheless achieve different absolute metric values.
7 Related work
Related work shows that prompt design affects software-engineering outcomes, but existing studies largely analyse strategies and templates rather than measurable linguistic properties.
- Prompt engineering in software engineering: Prompt engineering supports software-engineering tasks including code generation, testing, repository mining, startup support, and requirements engineering.In requirements engineering, studied tasks include elicitation, validation, traceability, and classification.
- Prompt engineering in software engineering: Existing SE research primarily treats prompts as task templates or strategy configurations such as zero-shot, few-shot, persona, and chain-of-thought prompting.These studies establish that prompt structure influences task performance but do not identify which linguistic properties matter.
- Linguistic analysis gap: SE studies rarely analyse prompts through syntax, morphosyntax, dependency patterns, lexical density, or surface complexity.This marks the linguistic-analysis gap addressed by the paper.
- Linguistic prompt analysis: NLP research has examined semantically equivalent prompts varying in mood, tense, aspect, modality, and synonym choice, finding model- and dataset-dependent performance variation.The supplied passage states that these differences are not reliably explained by prompt length, perplexity, word frequency, or ambiguity.
- This paper’s contribution: This paper treats prompts as measurable linguistic artefacts and links their features to LLM performance for software-engineering tasks.The identified predictors are intended for prompt selection, rewriting, and APE pre-filtering before costly inference.
8 Conclusions
The paper proposes linguistic profiling as an inexpensive, interpretable prior for prompt optimisation and outlines extensions across tasks, prompt components, models, and feature interactions.
- Practical implications: Linguistic profiling can score candidate prompts in milliseconds before LLM invocation, providing a fast pre-filter for expensive APE pipelines.The proposed use is to deprioritise structurally dense candidates early and reduce required LLM evaluations.
- Practical implications: This pre-filtering approach is particularly relevant to resource-constrained or energy-sensitive deployment contexts.The implication follows from reducing costly model evaluations before optimisation.
- Future directions: Future research should replicate the study across RE and SE tasks, expand manipulation to full prompt structure, and evaluate closed-source and larger-scale LLMs.The proposed prompt components include system context, task instruction, and output specification.
- Future directions: Future work should test joint linguistic-feature manipulation and interaction effects, then integrate predictors into hybrid APE and prompt-rewriting tools.The framework currently manipulates independent linguistic dimensions, while regression results suggest interactions are non-trivial.
Data Availability
The study’s data, prompts, model outputs, scripts, results, and instructions are available in an anonymised replication package.
- Replication package: An anonymised replication package contains the materials needed to reproduce the study and includes a README for the processing pipeline and analyses.The package includes data, prompts, model outputs, scripts, results, and instructions.