PaperlayerView exact revision on arXiv

Source-linked AI summary

MARQ: A Modular, LLM-Agnostic Requirement Quality Assessor Infused with Systems Engineering Knowledge

Jannatul Shefa, Alejandro Salado, Paul Wach, Taylan G. Topcu

arXiv:2610.36234v1cs.AIcs.SE

TL;DR

Off-the-shelf LLMs have shown missed detections, false alarms, and execution variability in requirement quality assessment, while engineering review remains dependent on expert judgment. MARQ infuses general-purpose LLMs with criterion-specific systems-engineering knowledge; it improved median F1 for every tested model, though the form of improvement varied by model.

  • Problem

    Off-the-shelf LLM assessment has shown missed defects, false alarms, uneven criterion performance, and execution variability, while requirement quality assessment remains dependent on expert judgment.

    Source: §1 · pp. 2–3

  • Method

    MARQ is a modular, LLM-agnostic prototype that infuses general-purpose LLMs with expert-derived, criterion-specific systems-engineering knowledge and is evaluated against off-the-shelf baselines across three models.

    Source: §1 · p. 3

  • Results

    MARQ improved median F1 for every tested model, improved Correctness issue detection for each, and brought a locally hostable 12B-parameter model to typical off-the-shelf frontier-model performance.

    Source: §5 · p. 23

  • Takeaways & Limitations

    SE knowledge infusion is a promising model-agnostic route to RE support, but evaluators should remain in human-in-the-loop workflows with final engineering judgment held by experts.

    Source: §5 · p. 23

    The evaluation uses a relatively small engineered system, so findings may not generalize to larger or more organizationally complex systems, other domains, or requirement sets with different maturity levels.

    Source: §5 · p. 23

Abstract

from arXiv · show

Requirements engineering (RE) is foundational to systems engineering (SE), as requirement defects can propagate downstream and adversely affect development outcomes. Although large language models (LLMs) are promising for requirement quality assessment, prior work shows that off-the-shelf models miss a large share of expert-identified defects, perform unevenly across quality criteria, and vary across executions. This study investigates whether infusing expert-derived SE knowledge can improve LLM-based requirement quality evaluation beyond off-the-shelf capability. We present MARQ, a modular, LLM-agnostic prototype that supplements LLMs with a criterion-specific knowledge stack operationalizing INCOSE requirement quality criteria and evaluated against off-the-shelf baselines on an expert-labeled NASA requirement set using Claude Opus 5, GPT-5.6 Sol, and the open-weight Gemma 3 12B, with 100 executions per model. MARQ achieves median F1 scores of 0.80 for Claude Opus 5, 0.62 for GPT-5.6 Sol, and 0.53 for Gemma 3 12B, compared with off-the-shelf baselines of 0.55, 0.52, and 0.18, respectively. However, the nature of these gains is model-dependent. Claude Opus 5 improves defect detection and false-alarm control; GPT-5.6 Sol gains detection and better execution-level consistency at the cost of more false alarms; and Gemma 3 12B reaches the performance of off-the-shelf proprietary models, although it remains highly variable. Criterion-level analysis shows gains concentrated in Correctness and Conformance, trade-offs between misses and false alarms within individual criteria, and model-dependent responses across several criteria. These findings position SE knowledge infusion as a promising model-agnostic pathway for strengthening trustworthy LLM-based RE support while underscoring the need for criterion-specific interventions and human-in-the-loop review.

1. Introduction and Background

Requirement defects can propagate into costly downstream rework, while quality assessment remains time-consuming and dependent on expert judgment. The section motivates MARQ as a way to test whether criterion-specific systems-engineering knowledge can improve LLM-based assessment.

1. Introduction and Background

  • Requirement errors can propagate through development, with correction costs rising from about 8× during design to more than 1,500× in operation.Quality assurance is critical but time-consuming and dependent on expert engineering judgment.Source: §1 · p. 2
  • AI-based RE research spans generation, analysis, translation, and quality assessment, with studies addressing both individual requirement defects and requirement-set issues.Individual-level work covers ambiguity, completeness, guideline adherence, and syntax; set-level work has focused on incompleteness, redundancy, conflict, and inconsistency.Source: §1 · pp. 2–3
  • Off-the-shelf LLMs show missed defects, false alarms, uneven criterion performance, and execution variability, motivating targeted knowledge integration.Prior benchmarking also found weak performance on criteria requiring contextual and systems-level judgment.Source: §1 · p. 3
  • Prior knowledge-augmentation studies largely supply information about the system, leaving less evidence on supplying knowledge about how engineers should judge quality criteria.The section distinguishes project or domain documentation from criterion-judgment knowledge.Source: §1 · p. 3
  • MARQ tests whether expert-derived, criterion-specific systems-engineering knowledge improves assessment beyond off-the-shelf LLM capability across model families and INCOSE criteria.The study asks how performance changes overall and how MARQ’s effects vary by criterion.Source: §1 · pp. 3–4
  • MARQ combines a reusable criterion-specific knowledge stack with a modular, LLM-agnostic architecture and evaluates it against off-the-shelf baselines across three models.The reported contributions include evidence on performance, error types, execution consistency, and criterion-level trade-offs.Source: §1 · pp. 3–4

2. Methodology

The methodology pairs an expert-consensus ground truth with a criterion-specific SE knowledge stack, then compares off-the-shelf and knowledge-infused LLM evaluations across repeated runs. It evaluates a NASA-derived requirement set against INCOSE criteria using a modular prototype and multiple performance metrics.

2.1 Step 1: Requirement Set Development

  • Thirteen requirements adapted from NASA’s Astrobee Simple Deployment Mechanism challenge form the evaluation set.The set was selected for its defined system context and balanced quality, and categorized as functional, performance, environmental, or resource requirements.Source: §2.1 · p. 5

2.2 Step 2: Ground Truth and Knowledge Base Generation

  • Three researchers independently assessed requirements against selected INCOSE criteria, then reconciled disagreements into a consensus ground truth.The panel included two seasoned reviewers with decades of experience; the resulting matrix contains 104 binary judgments across 13 requirements and eight criteria.Source: §2.2 · pp. 5–6

2.3 Step 3: The SE Knowledge Stack

  • The SE knowledge stack converts INCOSE criterion definitions into criterion-specific, testable rules based on reviewer reasoning and SE guidance.The rules are presented as checklists, with violation of any applicable subrule counting as a criterion violation; the stack and ground truth are independent artifacts sharing the panel’s interpretation.Source: §2.3 · pp. 6–7

2.4 Step 4: MARQ Prototype Architecture

  • MARQ separates context ingestion, knowledge-stack ingestion, prompt assembly, model adaptation, and evaluation visualization into five modules.Only the model adapter changes when switching among Claude Opus 5, GPT-5.6 Sol, and Gemma 3 12B.Source: §2.4 · p. 7, Figure 2 · p. 7

2.5 Step 5: Experimental Design and Data Collection

  • MARQ and off-the-shelf conditions differ only in whether the criterion-specific SE knowledge stack is supplied to the model.The evaluation objective, output format, and model settings are otherwise identical.Source: §2.5 · p. 8
  • Claude Opus 5, GPT-5.6 Sol, and Gemma 3 12B each ran 100 times in both conditions, producing 600 evaluation runs.No task-specific fine-tuning, retrieval augmentation, few-shot examples, or model-specific prompt optimization was used, isolating the added knowledge stack.Source: §2.5 · p. 8

2.6 Step 6: Data Analysis

  • Performance was assessed per run with seven metrics, prioritizing precision and F1 because class imbalance makes accuracy alone incomplete.The resulting distributions across 100 runs were examined with boxplots for central tendency, variability, spread, and outliers.Source: §2.6 · pp. 10–11

3. Findings

Across 100-run configurations, MARQ raises median F1 for all three models, but their gains differ in detection, false-alarm control, and execution-level variability. Criterion-level results further show that the direction and size of its effects depend on both model and quality criterion.

3.1 RQ1: Overall Effect of MARQ Relative to Off-the-Shelf Use

  • Median F1 rises from 0.55 to 0.80 for Claude Opus 5, 0.52 to 0.62 for GPT-5.6 Sol, and 0.18 to 0.53 for Gemma 3 12B.Table 2 summarizes seven metrics over 100 runs per configuration; Figures 5–7 show the model-specific performance distributions.Source: Table 2 · p. 11, §3.1 · p. 11, Figure 5 · p. 12

3.1.1 Claude Opus 5

  • Claude Opus 5’s median TPR rises from 0.63 to 0.83, while median FNR falls from 0.37 to 0.17 and median FPR from 0.26 to 0.11.Detection and false-alarm control improve together; across five of seven metrics, the worst MARQ run exceeds the best off-the-shelf run, while precision and F1 variability increase slightly.Source: §3.1.1 · pp. 11–12, Figure 5 · p. 12

3.1.2 GPT-5.6 Sol

  • GPT-5.6 Sol’s median TPR rises from 0.65 to 0.90 and its median FNR falls from 0.35 to 0.10, while median FPR increases from 0.34 to 0.41.Standard deviation decreases across every metric, including 75% for detection, but the detection gain comes with more unnecessary flags.Source: §3.1.2 · p. 13, Figure 6 · p. 13

3.1.3 Gemma 3 12B

  • Gemma 3 12B’s median F1 rises from 0.18 to 0.53, approaching the off-the-shelf proprietary models’ F1 scores of 0.55 and 0.52, but variability remains high.Median TPR rises from 0.17 to 0.57 and median FPR falls from 0.32 to 0.20, while variability increases for most aggregate metrics.Source: §3.1.3 · pp. 14–15, Figure 7 · p. 14

3.2.1 Type I Errors or False Flags (False Positive Rates) Across Quality Criteria

  • Criterion-level false-alarm effects vary across models: Claude improves notably on Conforming and Complete, GPT-5.6 Sol retains high false alarms, and Gemma improves on Necessary and Appropriate but worsens on Conforming.Claude’s Correct FPR rises as Correctness misses are eliminated; GPT’s Necessary and Correct overflagging contributes to its overall FPR increase, while Gemma’s Necessary and Appropriate gains are more variable.Source: §3.2.1 · pp. 15–18

3.2.2 Type II Errors or Misses (False Negative Rates) Across Quality Criteria

  • Criterion-level FNR effects also vary: Claude improves on Necessary and Correct but worsens on Complete and Singular, while GPT eliminates Necessary and Correct misses without notable deterioration elsewhere.For GPT, Correct FNR falls from a median of 0.67 to 0.00; Claude’s Complete and Singular medians each rise from 0.00 to 0.50.Source: §3.2.2 · pp. 18–19, Figure 11 · p. 19, Figure 12 · p. 20
  • Gemma’s Correct and Conforming FNRs improve, but Necessary, Complete, and Appropriate worsen; across all models, Correctness misses fall, with proprietary-model gains trading some false alarms for fewer misses.This within-criterion trade-off is not visible in aggregate metrics.Source: §3.2.2 · pp. 20–21, Figure 13 · p. 21

4. Discussion

The discussion evaluates MARQ against off-the-shelf use across three models and INCOSE criteria, emphasizing both performance gains and trade-offs. Results vary by model and criterion, supporting human review rather than treating the evaluator as uniformly reliable.

4.1 SE Knowledge Infusion Improves Every Model, but Not in the Same Way

  • Median F1 rises by 0.25 for Claude Opus 5, 0.10 for GPT-5.6 Sol, and 0.35 for Gemma 3 12B under MARQ.The authors describe SE knowledge as transferable across the three models, while noting that the nature and magnitude of gains differ.Source: §4.1 · p. 21, §4 · p. 21
  • Claude Opus 5 improves detection and false-alarm control, while GPT-5.6 Sol gains detection and consistency but incurs more false alarms.For GPT-5.6 Sol, standard deviations fall by 60–75% for the primary metrics; the authors caution that better detection alone does not mean better review support.Source: §4.1 · p. 22

4.2 A Locally Hosted Open-Weight Model Can Reach Off-the-Shelf Frontier Performance

  • Gemma 3 12B’s median F1 rises from 0.18 to 0.53, comparable to the off-the-shelf F1 of Claude Opus 5 (0.55) and GPT-5.6 Sol (0.52).Its execution-level variability remains extreme and widens under MARQ for most metrics, so the authors suggest aggregating repeated runs before relying on a single evaluation.Source: §4.2 · p. 22

4.3 Gains Are Concentrated in Specific Criteria and Involve Within-Criterion Trade-offs

  • Correctness is the clearest cross-model success, but proprietary models incur more false alarms; Correct and Conforming account for nearly two-thirds of ground-truth positives.The authors report that MARQ reduces Correctness misses for all three models, while false alarms rise from near zero to one-quarter and one-half of sound judgments for the proprietary models.Source: §4.3 · p. 22
  • Structural criteria such as Singular and Conforming may benefit more from rule-based or ontology-supported methods, while contextual criteria appear better suited to knowledge infusion.The authors note that MARQ’s modular architecture makes criterion-specific routing feasible.Source: §4.3 · p. 22

4.4 Implications for Trustworthy AI for SE

  • The authors position MARQ as an initial reviewer that targets expert attention, not a replacement for human validation and broader system-context judgment.They recommend judging evaluators by misses, false alarms, criterion-level behavior, and execution-level consistency, since one improved dimension can mask deterioration in another.Source: §4.4 · p. 23

5. Conclusion, Limitations, and Future Work

The conclusion reports improved median F1 for all three evaluated models, alongside model- and criterion-dependent trade-offs. It presents MARQ as a human-reviewed support tool and identifies limits in study scope and evaluation design, with further dataset, model, and human-AI studies proposed.

5. Conclusion

  • Across 600 runs, MARQ improved median F1 for every model, improved Correctness detection for every model, and brought a locally hostable 12B-parameter model to typical off-the-shelf frontier-model performance.The reported gains were concentrated in Correctness and Conformance, with within-criterion trade-offs or model-dependent deterioration on several criteria.Source: §5 · p. 23
  • The authors recommend human-in-the-loop use, with MARQ prioritizing reviewer attention while human experts validate flags and retain final engineering judgment.They also call for validation of evaluators in their intended roles, including within emerging agentic AI workflows.Source: §5 · p. 23

Limitations

  • The evaluation uses a relatively small engineered system and binary classification, limiting generalizability and excluding defect severity, explanation quality, and partially correct judgments.The authors caution that reported performance should not be treated as universally representative across AI-assisted requirement-quality settings.Source: §5 · p. 23
  • The study omits prompt engineering, retrieval augmentation, fine-tuning, domain-specific instruction tuning, and structured reasoning to isolate the effect of expert knowledge infusion.This design choice was intended to attribute performance differences as directly as possible to the added knowledge.Source: §5 · p. 24

Future Work

  • Future work should test larger, more diverse datasets and broader model families, explore richer criterion-specific knowledge and routing, and study human-AI review in practice.The authors also propose controls for prompt length and repeated-run aggregation to address execution-level variability.Source: §5 · p. 24

Electronic Appendix A: MARQ Prototype Repository

The authors report that the MARQ prototype and supporting documentation are available through the MARQ GitHub Repository.

Electronic Appendix A: MARQ Prototype Repository

  • The MARQ prototype and supporting documentation are available through the MARQ GitHub Repository.The authors state that the repository supports transparency, reproducibility, and further research.Source: Electronic Appendix A: MARQ Prototype… · p. 28

Related papers

Loading 2610.36234v1…