Two Truths and A Lie? Benchmarking Off-the-Shelf LLMs for Requirements Quality Assessment: Performance, False Alarms, and Misses

1. Introduction

• Requirement errors can become dramatically more expensive to correct later in the lifecycle, contributing to systems-engineering delays and cost overruns.Correction costs are reported at roughly 8× in design, 20× at manufacturing, 100× at integration and test, and over 1500× in operation relative to a requirements-phase fix.Source: §1 · p. 2
• The introduction situates LLM evaluation within expanding AI-for-systems-engineering research, while noting that its effectiveness for systems-engineering tasks remains an open question.It also outlines the paper’s literature review, methodology, findings, discussion, and conclusion.Source: §1 · pp. 2–3
• Evidence remains limited on LLMs’ reliability for engineering-requirement quality assessment, a task requiring contextual understanding and systems-engineering judgment.Prior work emphasized feasibility and appearance, with less attention to performance, error behavior, and evaluation reliability.Source: §1 · p. 2
• The study compares off-the-shelf LLM responses with expert-generated ground truth over independent trials and examines standard performance and error metrics.It asks how performance varies across INCOSE quality criteria and how it evolves across model generations.Source: §1 · pp. 2–3
• LLMs avoid false alarms better than they detect existing quality issues, with substantial variation across repeated evaluations and quality criteria.The findings support human-in-the-loop use rather than autonomous requirement quality evaluation.Source: §1 · p. 3

2.1 AI for RE (AI4RE)

• Prior AI4RE research spans generation, analysis, translation, and quality assessment, but many studies prioritize functionality over quality assessment.The review also describes NLP and transformer advances alongside varied AI4RE applications.Source: §2.1 · pp. 3–4

2.2 AI for Requirements Quality Assessment

• Requirement-quality studies address individual requirements and sets, but leave stochastic variability, error behavior, and cross-model performance open.Prior examples include ambiguity and defect detection for individual requirements, and redundancy, conflict, and inconsistency analysis for sets.Source: §2.2 · pp. 4–5

2.3 Research Gap

• Much AI4RE evidence comes from software requirements, leaving uncertain how well off-the-shelf LLMs evaluate engineered systems with physical, manufacturing, and operational constraints.The review notes that such issues may require deeper systems-level reasoning than software-only contexts.Source: §2.3 · p. 5
• Studies insufficiently compare established quality criteria, while inconsistent terminology makes findings difficult to compare or transfer across contexts.The review also notes limited coverage of all requirement-quality aspects.Source: §2.3 · p. 5
• Exploratory studies, fragmented benchmarks, inconsistent metrics, and limited coverage constrain reproducibility and trustworthiness; prior evidence also does not establish consistent gains across model generations.LLM outputs can vary across repeated executions, prompting conditions, and model versions, motivating evaluation of error distributions and variability.Source: §2.3 · pp. 5–6
• Evaluating individual LLM reliability is a form of component-level validation before embedding requirement-quality evaluators in larger agentic systems.The review describes these systems as decomposing complex tasks across specialized LLM-based agents or tool-specific modules.Source: §2.3 · p. 6
• The review identifies a first benchmark of off-the-shelf LLM requirement-quality assessment, covering overall performance, criterion-level errors, generation effects, and repeated-evaluation robustness.It frames this as a response to limited empirical evidence for engineered-system requirements.Source: §2.3 · p. 6

3. Methodology

• The five-step process adapts two requirement sets, establishes expert ground truth, defines experiments, automates repeated LLM evaluations, and compares outputs quantitatively.Figure 1 overviews the methodology.Source: §3 · p. 6, Figure 1 · p. 7

3.1 Step 1: Requirement Set Development

• Requirement Set A supplies the baseline; the less mature, poorer-quality Requirement Set B tests whether performance patterns generalize to a substantially different set.Set A contains 13 randomly selected requirements from the NASA-sponsored Simple Deployment Mechanism challenge; Set B contains 20 randomly selected requirements from a book-retrieval system exercise.Source: §3.1 · pp. 7–8

3.2 Step 2: Ground Truth Generation

• Three evaluators independently assessed requirements against nine INCOSE criteria, then discussed disagreements to reach a consensus ground truth.Feasibility and Verifiability were excluded because the available context lacked sufficient information about organizational capabilities.Source: §3.2 · pp. 8–9

3.3.1 Experiment 1: Single-Model Baseline Evaluation (RQ1 and RQ2)

• The baseline experiment evaluated Claude Opus 4.7 on Requirement Set A for 100 runs using its default sampling configuration.The model was selected after screening the latest Claude and GPT models because it performed better than the latest GPT model.Source: §3.3.1 · pp. 9–10

3.3.2 Experiment 2: Cross-Generational Model Comparison (RQ3)

• The cross-generational comparison tested five Claude and five GPT variants on Requirement Set A, with 100 runs per model and temperature 0.7 where configurable.Other pipeline configurations were held constant.Source: §3.3.2 · p. 10

3.3.3 Sensitivity Probe 1: Requirement Set Robustness; 3.3.4 Sensitivity Probe 2: Temperature Robustness

• Two sensitivity probes tested generalizability on Requirement Set B and temperature effects on Requirement Set A using GPT-5.4 across five settings and 100 runs per setting.The temperature settings were 0.0, 0.3, 0.5, 0.7, and 1.0; the other configurations were otherwise identical.Source: §3.3.3 · p. 10, §3.3.4 · p. 10

3.4 Step 4: LLM Data Collection

• An automated n8n workflow used zero-shot prompts with requirement context, parsed structured model responses, and stored both evaluation matrices and raw outputs.Inputs included the requirements, problem description, and ConOps; the parsed output was transformed into a binary evaluation matrix.Source: §3.4 · pp. 11–12

3.5 Step 5: Data Analysis

• Each run was compared with expert ground truth using normalized metrics; precision and F1 were prioritized because class imbalance can make accuracy misleading.For Set A, quality violations occupied approximately 26% of ground-truth cells, so a model flagging no issues could still reach 74% accuracy.Source: §3.5 · pp. 12–13

4.1 RQ 1: How well does an off-the-shelf LLM perform in terms of evaluation of requirements quality?

• 0.33 median TPR and 0.67 median FNR contrast with 0.89 median TNR and 0.11 median FPR for Claude Opus 4.7.Median Accuracy is 0.74, but the authors caution that it reflects stronger true-negative than true-positive performance; median Precision is 0.48.Source: Figure 3 · p. 14, §4.1 · pp. 14–15
• For Claude Opus 4.7, median F1-score is 0.40, and none of the evaluation metrics approaches perfect performance, even in the best trials.The F1-score ranges approximately from 0.17 to 0.57 across executions.Source: §4.1 · p. 15

4.2 RQ 2: How do error rates vary for various INCOSE Requirements Engineering quality criteria?

• Criterion-level analysis focuses on FPR and FNR, while some metrics have no plotted distribution when the relevant ground-truth instances are absent.No Issues and Unsure of Category have undefined FNR because the denominator is zero; a collapsed Necessity FNR at 1.00 means issues were never detected.Source: §4.2 · p. 15

4.2.1 Type I Errors or False Flags (False Positive Rates) Per Quality Criterion

• Conformance has the highest false-alarm tendency, while Completeness, Necessity, and Singularity show comparatively moderate FPRs.Correctness and Appropriateness have FPR distributions collapsed at 0.00; Conformance FPR has a median of 0.67 and Ambiguity a median of 0.33.Source: §4.2.1 · pp. 15–16, Figure 4 · p. 16

4.2.2 Type II Errors or Misses (False Negative Rates) Per Quality Criterion

• Appropriateness, Unambiguity, Completeness, Singularity, and Conformance each have median FNR 0.50, but their distributions differ substantially.Completeness FNR ranges from 0.00 to 1.00 with standard deviation 0.39; Unambiguity has mean FNR 0.43 and standard deviation 0.12.Source: §4.2.2 · pp. 17–18
• Necessity issues are always missed, and Correctness issues are almost always missed, with median FNRs of 1.00 and 0.89, respectively.The authors advise caution when using off-the-shelf LLMs to assess these two criteria.Source: Figure 5 · p. 17, §4.2.2 · p. 17

4.3 RQ3: How does off-the-shelf LLM performance for RE evolve over-generations?

• Across Claude and GPT generations, performance is non-monotonic: GPT-5 reaches median F1 0.43, then later variants fall to 0.31, 0.37, and 0.30.Claude accuracy broadly rises to 0.74 in Opus 4.7, yet later models do not consistently produce more stable outcomes across runs.Source: §4.3 · pp. 18, 21, 22, Figure 6 · p. 19, Figure 7 · p. 21

4.4.1 Sensitivity Probe 1: Requirement Set Robustness

• Set B exceeds Set A on median TPR across all ten models, with higher precision and F1 for every model, while accuracy falls for all Claude models and GPT-4o.The largest TPR gains include GPT-5 (0.47 to 0.61) and Sonnet 4.6 (0.27 to 0.40); the accuracy exception is limited to those models.Source: Figure 8 · p. 23, Figure 9 · p. 24, §4.4.1 · p. 25
• Across Requirement Set B, generational patterns remain non-monotonic: GPT-5 leads in median TPR and F1, while later GPT and Claude variants decline on some metrics.GPT-5 has median TPR 0.61 and F1 0.75; GPT-5.4 and GPT-5.5 have TPR 0.42 and 0.53 and F1 0.58 and 0.68.Source: Figure 8 · p. 23, Figure 9 · p. 24, §4.4.1 · p. 24
• Execution-level variability also persists across requirement sets, with GPT-5.1 especially unstable and GPT-5.4 comparatively stable on Set B.GPT-5.1 has TPR range 0.36–0.93 (std=0.153), while GPT-5.4 has TPR and F1 standard deviations of 0.021.Source: §4.4.1 · p. 25

4.4.2 Sensitivity Probe 2: Temperature Robustness

• For GPT-5.4 on Set A, performance metrics vary modestly across temperatures without a consistent trend, and run-to-run variability persists even at low temperatures.Median TPR ranges from 0.32 to 0.40 across the five settings; median F1 ranges from 0.32 to 0.37.Source: §4.4.2 · pp. 25, 27, Figure 10 · p. 26

5.1 Missed Detections Are the Dominant Risk in LLM-Based Requirement Evaluation

• Missed detections dominate false alarms, limiting dependable autonomous use; the authors instead support human-in-the-loop decision support.Missed defects may propagate into downstream engineering work, while false alarms can prompt unnecessary revisions and verification activities.Source: §5.1 · p. 28, §5 · p. 28

5.2 Requirement Quality Criteria Are Not Equally Suitable for LLM-Based Evaluation

• Performance differs by criterion: Unambiguous and Conforming issues are more detectable, while Correct, Necessary, and Complete issues often go undetected.The discussion links correctness and necessity judgments to integrating the problem statement, ConOps, and broader systems-engineering knowledge.Source: §5.2 · pp. 28–29

5.3 Newer LLM Generations Do Not Necessarily Yield Better Requirement Evaluation Performance

• Newer generations do not consistently outperform predecessors, and gains in one metric can accompany declines in another.For example, Sonnet 4.6 improves TPR over Opus 4.5 but has higher FPR; GPT-5.5 has higher TNR than GPT-5.4 but lower accuracy and precision.Source: §5.3 · pp. 29–30

5.4 Temperature Tuning Does Not Guarantee Improved Evaluation Performance or Stability

• Within the examined temperature range, tuning neither substantially changes typical performance nor reliably controls execution-level variability.Variability persists at low temperatures, including 0.0 and 0.3.Source: §5.4 · p. 30

5.5 Limitations

• The benchmark uses a point-mass manipulator, constraining generalizability to other domains, requirement maturity levels, and more complex systems.The authors characterize the results as a conservative benchmark and expect documented issues to worsen with greater scope and complexity.Source: §5.5 · p. 30
• Results characterize off-the-shelf models under a defined prompting setup, not optimized performance after specialized prompting or adaptation.The study does not exhaustively cover available model versions, uses a limited temperature range, and treats assessment as binary classification, omitting severity, rationale quality, and partial correctness.Source: §5.5 · pp. 30–31

6. Conclusion

• Off-the-shelf LLMs are not yet trustworthy for autonomous requirement evaluation, so the authors recommend decision support that preserves human authority over final assessments.Observed execution variability also makes single-execution evaluations statistically unreliable, especially for high-stakes requirement sets.Source: §6 · p. 31
• Future work should target reasoning and engineering judgment, including prompt engineering, INCOSE-grounded examples, or retrieval-augmented context; Correct and Necessary are priority criteria.The conclusion also calls for human–AI collaborative workflows that balance automated support with expert judgment.Source: §6 · p. 31
• Newer generations do not consistently outperform predecessors, and temperature tuning does not reliably improve performance or reduce variability.The conclusion frames AI-assisted evaluation as augmenting rather than replacing systems engineers.Source: §6 · p. 31

Appendix A

• The prompt asks for individual requirement evaluation using project context, ConOps information, and a reference image.It instructs the evaluator to assess alignment with project goals, needs, and operational constraints.Source: Appendix A · p. 38
• The prompt distinguishes A1 as “No issues” and A2–A9 as criteria whose violations should be reported.For each issue, it requests the criterion ID and name plus a short explanation.Source: Appendix A · p. 38
• The rubric covers necessity, appropriateness, ambiguity, completeness, singularity, correctness, conformance, and uncertainty of category.The prompt directs the evaluator to report each violated criterion and to mark A1 only when the requirement has no problems.Source: Appendix A · p. 38

Appendix B

• Appendix B presents supplementary criterion-level analysis of error rates.Source: Appendix B · p. 39
• The analysis reports distributions for TPR, TNR, accuracy, precision, and F1.Source: Appendix B · p. 39
• The appendix uses Claude Opus 4.7 and provides brief descriptions of each metric’s criterion-level analysis.Source: Appendix B · p. 39

Appendix B.1. True Positive Rate (TPR) Per Quality Criterion

• Necessary has a TPR of 0.00 across executions, while Correct has a median TPR of 0.11 and mean of 0.09.The figure presents TPR distributions by quality criterion.Source: Appendix B.1 · p. 39, Figure B1 · p. 40
• Appropriate, Singular, and Complete each have a median TPR of 0.50, but their detection is unstable across executions.Appropriate has a mean TPR of 0.29; Singular and Complete each range from 0.00 to 1.00, and Complete has the largest standard deviation.Source: Appendix B.1 · p. 40
• Unambiguous and Conforming each have a median TPR of 0.50, while No Issues and Unsure of Category are omitted because TPR is undefined without true positives.Unambiguous has a mean TPR of 0.57 and ranges from 0.25 to 0.75; Conforming ranges from 0.10 to 0.80.Source: Appendix B.1 · p. 40

Appendix B.2. True Negative Rate (TNR) Per Quality Criterion

• TNR values are substantially higher than TPR values, indicating that the model is generally stronger at avoiding false alarms than detecting actual quality issues.Figure B2 presents TNR distributions by issue category.Source: Appendix B.2 · p. 40, Figure B2 · p. 41
• Conforming has the weakest TNR, with a median of 0.33 and mean of 0.55; Unambiguous has a median of 0.67 and mean near 0.70.Conforming ranges from approximately 0.33 to 1.00, while Unambiguous ranges from approximately 0.22 to 0.89.Source: Appendix B.2 · p. 41
• Appropriate, Correct, and Unsure of Category have TNR distributions collapsed at 1.00, while No Issues has a median TNR of 0.92.Necessary, Singular, and Complete have median TNR values of 0.83, 0.82, and 0.91, respectively.Source: Appendix B.2 · p. 41

Appendix B.3. Accuracy Per Quality Criterion

• Correct has a median accuracy of 0.38 and mean of 0.37, while Conforming has a median of 0.46 and mean of 0.47.Correct is consistently low with little variability; Conforming ranges from approximately 0.15 to 0.80.Source: Appendix B.3 · p. 41, Figure B3 · p. 42
• Appropriate has a median accuracy of 0.85 and mean of 0.88, No Issues has a median of 0.92 and mean of 0.89, and Unsure of Category is at 1.00 across executions.The figure provides accuracy distributions by quality criterion.Source: Figure B3 · p. 42, Appendix B.3 · p. 42
• Unambiguous has a median accuracy of 0.69 and mean of 0.66, while Singular, Complete, and Necessary each have median accuracy around 0.77.Unambiguous shows noticeable variability, and Complete and Singular also show moderate variability.Source: Appendix B.3 · p. 42

• Precision is 0.00 for No Issues and Necessary, so positive flags in both categories are consistently incorrect.Source: Appendix B.4 · p. 42, Figure B4 · p. 43
• Complete has median Precision 0.33 and mean 0.46, with values spanning 0.00 to 1.00.Source: Appendix B.4 · p. 42, Figure B4 · p. 43
• Singular has median Precision 0.29 and mean 0.21, indicating that singularity-related issue flags are often unreliable.Source: Appendix B.4 · p. 42
• Unambiguous Precision centers around 0.50 with comparatively stable spread, while Conforming has median Precision 0.71 and mean 0.81.Source: Appendix B.4 · p. 42
• Appropriate and Correct both have median Precision 1.00, but Appropriate ranges from 0.00 to 1.00 while Correct remains at 1.00.Source: Appendix B.4 · p. 43, Figure B4 · p. 43
• Correctness-related flags are valid when made, but earlier TPR results show correctness issues are rarely detected.Source: Appendix B.4 · p. 43

• Correct F1 remains 0.20 across all executions, indicating consistently poor balance between issue detection and reliable positive predictions.Source: Appendix B.5 · p. 43, Figure B5 · p. 44
• Singular has median F1-score 0.40 and mean 0.44, with limited issue detection despite occasional stronger executions.Source: Appendix B.5 · p. 43
• Appropriate has the strongest overall F1, with median 0.67 and mean 0.64; Conforming has median 0.59 and mean 0.53.Source: Appendix B.5 · p. 44, Figure B5 · p. 44
• Unambiguous and Complete both have median and mean F1-scores of 0.50, but Complete varies more across executions.Source: Appendix B.5 · p. 44
• No Issues, Necessary, and Unsure of Category are omitted because F1 is undefined; Necessary had no true positives across executions.Source: Appendix B.5 · p. 44, Figure B5 · p. 44