PaperlayerView exact revision on arXiv

Source-linked AI summary

Does ISO-Grounded NFR Specification Improve LLM Code Generation? A Comparison of Rich and Structured Interventions against a Natural-Language Baseline

Joào Pedro Monteiro Pereira, Vinicius Cardoso Garcia

arXiv:2608.13742v1cs.SEcs.AIcs.LG

TL;DR

It remains unclear whether ISO-grounded NFR enrichment helps LLMs generate better code, or whether prose-versus-JSON format matters more. This study compares both interventions with terse one-line NFRs and finds that enrichment improves quality proxies and robustness, but not functional correctness, while prose and JSON perform similarly when content is fixed.

  • Problem

    It remains unclear whether enriching terse NFRs with ISO/IEC 25010 improves LLM-generated code, or whether prose-versus-JSON format matters more.

  • Method

    The study compares ISO-grounded rich prose and equivalent JSON against one-line NFRs across four NFRs, ten prompt variations, and HumanEval/HumanEval-ET.

  • Results

    ISO-grounded enrichment improves quality proxies and robustness but does not reliably improve functional correctness; prose and JSON differ negligibly when content is fixed.

  • Takeaways & Limitations

    What NFRs specify—standard-grounded enrichment—matters more for quality and robustness than whether the content is serialized as prose or JSON.

  • Takeaways & Limitations

    Correctness comparisons are threatened by non-concurrent data collection because the NL-simple baseline predates the interventions by several weeks.

Abstract

from arXiv · show

In LLM-based code generation, Non-Functional Requirements (NFRs) are often specified as terse one-line phrases. We ask whether grounding those specifications in ISO/IEC 25010 Quality Model, either as rich natural-language prose (NL-rich) or as structured JSON (Structured), improves code generated on HumanEval/HumanEval-ET compared to a RobuNFR-style one-line baseline (NL-simple). We evaluate four NFRs (performance, error handling, code smell, readability) with ten prompt variations per condition under a fixed model snapshot and paired non-parametric analysis. Primary finding: ISO-grounded enrichment improves static quality proxies (unreadability density falls across all four NFRs (e.g., Performance 0.88 -> 0.69 for NL-rich)) and reduces sensitivity to prompt wording, but does not reliably improve functional correctness; for error handling, extended-test pass rate decreases, suggesting tension between defensive coding patterns and exact-output benchmarks. Secondary finding: when ISO content is held constant, NL-rich and Structured differ negligibly in correctness (|delta| <= 0.023), indicating that semantic content matters more than JSON-vs-prose format. Practitioners should invest in standard-grounded NFR content rather than serialization form. A fully traceable replication package is provided.

1 Introduction

The study tests whether ISO/IEC 25010:2023-grounded NFR enrichment improves LLM-generated code beyond terse NL-simple specifications, and whether prose or JSON representation matters more. It compares correctness, NFR-specific quality, prompt robustness, and representation form under controlled conditions.

  • Problem: NFRs are often short and ambiguous, motivating comparison with ISO/IEC 25010:2023-grounded specifications.The baseline follows RobuNFR’s one-line-per-NFR style.
  • Preview of answer: With ISO content held constant, prose and JSON behave similarly, indicating that specified content matters more than serialization form.This is the study’s secondary representation-form comparison.
  • Investigation: The study compares NL-rich detailed prose and Structured JSON against a RobuNFR-style NL-simple baseline.All conditions use gpt-5.4-2026-03-05, with ten prompt variations per condition.
  • Preview of answer: Enrichment improves quality proxies and robustness relative to NL-simple, but not functional correctness, and may harm Error Handling ET-Pass@1.The comparison also examines Pass@1, ET-Pass@1, NFR-specific densities, and sensitivity across prompt variations.
  • Contributions: The contributions include paired intervention-baseline comparisons, a content-controlled form comparison, evidence of narrower prompt sensitivity, and a traceable replication package.The package links every reported number to its source file.

2 Background

The study grounds NFR content in ISO/IEC 25010:2023 and evaluates generated code with functional-correctness measures plus normalized static and lightweight dynamic quality proxies. These proxies target selected performance, reliability, and maintainability concerns but do not cover all ISO/IEC 25010 characteristics.

  • Standards grounding: ISO/IEC 25010:2023 defines product-quality characteristics including Performance Efficiency, Reliability, and Maintainability, while ISO/IEC 25002:2024 provides usage guidance.The standards serve as an external, citable source of NFR content.
  • Evaluation measures: Pass@1 on HumanEval and MBPP measures functional correctness, while ET-Pass@1 on HumanEval-ET extended oracles reduces false positives.The evaluation reports both Pass@1 and ET-Pass@1 because original test suites can be weak.
  • NFR proxies: Static analysis reports code-smell density and unreadability density per ten LOC, exception density per ten LOC, and mean execution time.Per-LOC normalization accounts for richer or structured prompts producing longer code.
  • Proxy rationale: The selected proxies are automatable across 164×10 completions per condition and map reasonably to ISO wording while remaining comparable to RobuNFR reporting.They focus on performance efficiency, fault tolerance, and maintainability-related concerns that admit static or lightweight dynamic signals.
  • Scope and limitations: The four NFR-focused proxies do not measure all ISO/IEC 25010 characteristics, including security and compatibility.The selected concerns were limited to those suitable for static or lightweight dynamic signals on HumanEval.

3 Related Work

Prior work shows that NFR phrasing, prompt variation, and ISO quality models influence LLM-generated code, but leaves open whether ISO-grounded enrichment or input format changes outcomes under controlled comparison. This study addresses that gap with paired enrichment and content-controlled form contrasts.

  • Robustness of NFR-aware generation: RobuNFR finds that NFR inclusion can reduce Pass@1 by up to 39% and increase variability across prompt rewordings, using intentionally terse one-line NFR prompts.Its protocol evaluates four NFR dimensions through prompt variation, regression testing, and diverse workflows.
  • Requirement-aware generation: ArchCode organizes functional and non-functional requirements and introduces HumanEval-NFR, whereas this study examines how NFR specification depth changes outcomes for fixed functional tasks.The distinction is between requirement organization and controlled manipulation of NFR content.
  • Structured vs. natural-language requirements: When ISO content is fixed, the related-work hypothesis that LLMs might prefer structured input receives no correctness advantage for format alone.This motivates the direct comparison of structured and natural-language requirements under content control.
  • NFR specification quality and ambiguity: ISO/IEC 25010 provides a relevant backbone for explicit NFRs, while prior work does not test enrichment against a one-line baseline or hold ISO content constant while varying form.Requirements-engineering studies also identify ambiguity and imprecision as recurring NFR problems.
  • Research gap: The study fills a gap in paired, per-problem comparisons of NL-simple versus ISO-grounded interventions and in content-controlled contrasts between input forms.Prior work establishes that NFRs affect outputs, prompt variation matters, and ISO quality models are relevant.

4 Research Method · 4.1 Overview and Experimental Conditions · 4.2 Illustrative Prompt Contrast (Code Smell NFR)

The study compares NL-rich and Structured ISO/IEC 25010 interventions with a RobuNFR-style NL-simple baseline under controlled generation and evaluation conditions. An illustrative Code Smell prompt shows that enrichment adds explicit ISO characteristics, constraints, and acceptance criteria while preserving the functional task clause.

  • 4.1 Overview and Experimental Conditions: NL-simple is the RobuNFR-style one-line NFR baseline, while NL-rich uses ISO/IEC 25010 prose and Structured serializes the same content as JSON.Structured JSON includes the attribute, intent, ISO mapping, constraints, and acceptance criteria.
  • 4.1 Overview and Experimental Conditions: NL-rich and Structured differ only in representation form because they share identical ISO-grounded content.Both interventions use the same ISO content; the comparison isolates prose versus JSON serialization.
  • 4.1 Overview and Experimental Conditions: All three primary conditions use gpt-5.4-2026-03-05, temperature 0, the HumanEval problem set, and the same evaluation pipeline.The study also reports a Function-Only no-NFR baseline as an NFR-independent context row.
  • 4.1 Overview and Experimental Conditions: The design contrasts each intervention with NL-simple for RQ1–RQ3 and compares NL-rich with Structured under fixed ISO content for RQ4.Both comparison axes reuse RobuNFR’s ten-variation protocol and the same functional task clause.
  • 4.2 Illustrative Prompt Contrast (Code Smell NFR): The Code Smell illustration presents three NFR blocks whose surface wording varies across ten prompt templates, while ISO fields remain identical between NL-rich and Structured.The example makes the experimental manipulation concrete for one quality attribute.
  • 4.2 Illustrative Prompt Contrast (Code Smell NFR): The functional prefix, “complete the following code:” plus the HumanEval stub, is identical across all conditions.This keeps the coding task constant while varying the NFR block.
  • 4.2 Illustrative Prompt Contrast (Code Smell NFR): NL-simple leaves interpretation to the model, whereas NL-rich and Structured specify ISO characteristics, explicit constraints, and quality-attribute acceptance criteria.The acceptance criteria are tied to the quality attribute rather than to passing HumanEval tests.

4.3 Holding Content Constant; Varying Only Form … 4.6 Snapshot Stability Check (Batch Mitigation)

The study isolates serialization form by holding ISO-grounded NFR content and task wording constant, while separate-batch stability checks assess whether snapshot drift threatens comparisons. The full-set rerun found negligible Performance drift, although a small exploratory subset showed a significant Performance difference.

  • 4.3 Holding Content Constant; Varying Only Form: NL-rich and Structured use one shared ISO object per NFR, with identical functional-task wording; only prose versus JSON varies.The shared object contains the attribute, intent, ISO characteristic/sub-characteristics, constraints, and acceptance criteria.
  • 4.4 ISO Mapping: Performance maps to ISO/IEC 25010 Performance Efficiency, error handling to Reliability, and code smell/readability to Maintainability.The mappings use time behavior and resource utilization, fault tolerance, and maintainability sub-characteristics respectively.
  • 4.4 ISO Mapping: Each shared ISO object contains an aligned intent, two to four imperative constraints, and attribute-specific acceptance criteria; ten variations paraphrase surface wording.NL-rich renders these fields as connected prose, whereas Structured emits the same fields as JSON keys.
  • 4.5 Model, Benchmark, and Generation: All conditions use gpt-5.4-2026-03-05 at temperature 0, but NL-simple was collected in April–May 2026 and interventions in June 2026.The differing batch timing motivates separate post hoc snapshot-stability checks.
  • 4.6 Snapshot Stability Check (Batch Mitigation): 0.93 versus 0.92 mean Performance Pass@1, 97.6% per-task agreement, p=0.424, and Cliff’s δ=0.01 indicated negligible full-set drift.The August 2026 rerun covered all 164 HumanEval tasks; the bootstrap 95% CI was [−0.04, +0.01].
  • 4.6 Snapshot Stability Check (Batch Mitigation): 0.84 versus 0.84 ET-Pass@1 was non-significant (p=0.773) in the full-set stability check.This result further weakens evidence that snapshot drift affected the complete-set comparison.
  • 4.6 Snapshot Stability Check (Batch Mitigation): 0.63 versus 0.77 Performance Pass@1 on the 30-task pilot subset had p=0.072 and a bootstrap CI of [+0.03, +0.27], while Error Handling was stable at 0.83 versus 0.87 (p=1.0).The subset over-sampled failing tasks, so its absolute Pass@1 rates were lower than full-baseline rates.
  • 4.6 Snapshot Stability Check (Batch Mitigation): Non-concurrent collection remains an internal-validity concern, but the August full-set rerun suggests Performance snapshot drift is unlikely to explain the primary comparisons.The study treats the pilot subset as underpowered exploratory evidence and retains original batch timestamps for primary statistics.

4.7 Metrics · 4.8 Statistical Analysis · 4.9 Reproducibility

The study evaluates functional correctness, NFR-quality, size, and execution-time metrics with paired non-parametric comparisons, while releasing artifacts and traceability data for independent verification. Robustness analyses compare prompt conditions using standardized per-problem density vectors and aggregate-metric variability.

  • 4.7 Metrics: The reported metrics include Pass@1, ET-Pass@1, exception density, code-smell density, unreadability density, LOC, and execution time.Density metrics count issues per ten LOC; execution time is averaged over five runs per problem and its ET variant.
  • 4.7 Metrics: NFR-quality metrics are compared on a shared set of problems with per-problem pairing to avoid survivorship bias.The pairing is specified as part of the Section 4.8 analysis design.
  • 4.8 Statistical Analysis: NL-rich and Structured are each contrasted with NL-simple using paired Wilcoxon signed-rank tests, Cliff’s delta, and Holm–Bonferroni correction.Corrections are applied within each NFR, comparison, and metric family; δ > 0 favors the intervention.
  • 4.8 Statistical Analysis: The non-parametric paired design addresses bounded, often-skewed Pass@1 and density metrics while respecting task-level pairing without assuming normality.The same HumanEval task is evaluated under the two prompt conditions.
  • 4.8 Statistical Analysis: The secondary RQ4 contrast compares NL-rich with Structured while holding content constant, using baseline and intervention vectors from specified evaluation runs.Baseline vectors come from NL-simple evaluation JSONs, while intervention vectors come from the June 2026 runs.
  • 4.8 Statistical Analysis: Per-problem unreadability, code-smell, and exception density vectors are derived from Pylint buckets and exception statements normalized by LOC.The passage also identifies aggregate-metric variability across ten prompt conditions as the robustness analysis reported in Table 5.
  • 4.9 Reproducibility: All configurations, prompts, generated code, evaluation outputs, and analysis scripts are released in a replication package.Required execution-time changes are documented in RUNTIME_NOTES.md.
  • 4.9 Reproducibility: results_numbers.json links every table cell and p-value to its originating evaluation JSON or Excel summary for independent verification without rerunning generation.This traceability mapping supports checking reported results against their source artifacts.

5 Results

ISO-grounded enrichment improves static-quality densities and stability across NFRs, but does not reliably improve functional correctness and can reduce Error Handling extended-test performance. With ISO content fixed, NL-rich and Structured are effectively equivalent for correctness, making content more important than serialization form.

  • Functional correctness: For Performance, Code Smell, and Readability, intervention Pass@1 and ET-Pass@1 effects are negligible and non-significant after Holm correction.Aggregate mean Pass@1 differs by at most 1.4 percentage points from NL-simple, including Performance NL-rich 94.5 versus NL-simple 93.1.
  • Functional correctness: Error Handling NL-rich ET-Pass@1 drops from 81.0 to 76.0 with pHolm= 0.001, while Structured drops to 78.4.NL-rich Pass@1 also falls from 95.1 to 92.3; the passage characterizes the extended-test decrease as modest but systematic.
  • Static quality: Unreadability density falls for every NFR under both interventions, with all eight comparisons significant after Holm correction.Performance declines from 0.88 to 0.69 for NL-rich and 0.79 for Structured; Error Handling declines from 0.57 to 0.42/0.43.
  • Robustness: Unreadability STDEV falls from 0.123 to approximately 0.037 for Performance, with all eight intervention–baseline contrasts significant after Holm correction.Intervention prompts are also less textually diverse than NL-simple, with Jaccard 0.10–0.13 versus 0.44–0.59, yet show lower unreadability sensitivity.
  • Representation form: When ISO content is held constant, every Structured-versus-NL-rich Pass@1 and ET-Pass@1 comparison has negligible effect size (|𝛿| ≤0.023) and is non-significant after Holm correction.Aggregate unreadability also differs only slightly between forms, such as Performance 0.69 for NL-rich versus 0.79 for Structured.

6 Discussion

ISO-grounded enrichment improves static-quality proxies and robustness relative to terse NFRs, but generally leaves functional correctness unchanged and can reduce error-handling benchmark performance. When ISO content is held constant, JSON and prose produce negligible correctness differences, so format is secondary to content and workflow needs.

  • Quality and robustness: For three of four NFRs, correctness effects are negligible, while Error Handling significantly reduces ET-Pass@1 and increases exception density.The three unaffected NFRs are Performance, Code Smell, and Readability; Error Handling has p_Holm = 0.001 for NL-rich and 0.048 for Structured.
  • Quality and robustness: ISO-grounded enrichment lowers unreadability density across all four NFRs and reduces code-smell density when Code Smell is targeted.Performance unreadability falls from 0.88 for NL-simple to 0.69 for NL-rich; Code Smell density drops from 0.019 to 0.011 for NL-rich.
  • Quality and robustness: Enrichment lowers unreadability STDEV for every intervention–baseline pair and reduces prompt diversity relative to NL-simple, although Pass@1 STDEV gains are not inferentially confirmed.Mean pairwise Jaccard distance is approximately 0.10–0.13 for interventions versus 0.44–0.59 for NL-simple.
  • Representation form: JSON and prose do not differ significantly in correctness when ISO fields are identical, while aggregate quality-density differences remain slight.All Pass@1 and ET-Pass@1 contrasts have negligible Cliff’s δ and fail after Holm correction; Performance unreadability is 0.69 versus 0.79.
  • Scope and implications: The benchmark observes function-level static quality and exact-output correctness, so the findings do not establish system-level maintainability or reliability.HumanEval’s small, self-contained Python functions are well suited to functional correctness but weakly sensitive to most ISO/IEC 25010 characteristics.
  • Scope and implications: Practitioners should choose ISO-grounded enrichment for cleaner, more stable maintainability-oriented code, select JSON or prose for integration needs, and validate reliability with project oracles.For maximizing HumanEval pass rate, enriched Error Handling may hurt ET-Pass@1, whereas enrichment is preferable for reducing static issues or stabilizing outputs.

7 Threats to Validity

The study’s validity is limited by proxy-based quality measures, non-concurrent data collection, and a narrow evaluation setting. Statistical controls and fixed experimental conditions mitigate some threats, while the findings remain bounded to this benchmark, model snapshot, and function-level analysis.

  • Construct validity: Static-analysis proxies and exception density do not capture runtime reliability, security, or user-perceived maintainability.Exception-statement density may reflect broad or low-quality try/except blocks rather than improved fault tolerance.
  • Internal validity: The NL-simple baseline predates NL-rich and Structured by several weeks, creating a non-concurrent data-collection threat.The study fixes the model snapshot, temperature 0, benchmark, and evaluation pipeline across conditions to limit API drift.
  • Conclusion validity: Correctness, time, and density metrics use per-problem paired vectors with n=164 and non-parametric paired testing.The analysis reports Cliff’s delta and applies Holm–Bonferroni correction separately within each NFR, comparison, and metric family.
  • External validity: The findings use one model, one HumanEval/ET benchmark, and four NFRs, limiting generalization to other models and repository-level or multi-file tasks.HumanEval’s small single-function tasks and exact-output oracles are weakly sensitive to system-level ISO/IEC 25010 characteristics.
  • Reliability/Reproducibility: Execution time depends on hardware, so the study compares only internal deltas within the same experimental run.The full pipeline and data are released in a replication package with results_numbers.json traceability from tables to source files.

8 Conclusion and Future Work

ISO-grounded NFR enrichment improves static-analysis quality densities and reduces prompt sensitivity, but does not reliably improve functional correctness. Prose and JSON serialization have negligible correctness differences when content is held constant, while several replication and evaluation extensions remain future work.

  • Conclusion: ISO-grounded enrichment improves static-analysis quality densities and reduces prompt sensitivity versus one-line NFRs, but does not reliably improve functional correctness.For Error Handling, enrichment may reduce extended-test pass rate.
  • Conclusion: When content is held constant, prose versus JSON form has negligible effect on correctness.The comparison is between NL-rich prose and Structured JSON interventions grounded in ISO/IEC 25010:2023.
  • Future Work: Future work includes MBPP replication, additional models and decoding settings, human evaluation of ISO-aligned quality, and direct measurement of requirement ambiguity.Architecture-centric benchmarks such as HumanEval-NFR would test transfer when functional and non-functional requirements are co-specified.

Artifact Availability

A fully traceable replication package is available under an MIT License, containing prompt configurations, generated completions, evaluation outputs, analysis scripts, and documentation. It also includes result-traceability files and W1 snapshot-stability checks.

  • Artifact Availability: The MIT-licensed replication package provides prompt configurations, generated completions, evaluation outputs, analysis scripts, and documentation.It is available at https://doi.org/10.5281/zenodo.21880022.
  • Artifact Availability: results_numbers.json supports table and p-value traceability, while rq3_per_problem_stdev.json contains RQ3 per-problem STDEV inferential tests.
  • Artifact Availability: W1 snapshot-stability checks include a pilot with n=30 and a Performance check with n=164.The files are stored under results/w1-stability.

A ISO Prompt Object Schema (Replication)

NL-rich and Structured prompts are generated from a shared JSON object for each NFR and variation, differing only in serialization. The schema encodes ISO-aligned intent, scoped constraints, and test-independent acceptance criteria.

  • Shared prompt object: Each NL-rich and Structured prompt is generated from one JSON object per NFR and variation, with identical keys across forms and serialization as the only difference.This design isolates JSON-versus-prose serialization while holding prompt content constant.
  • Schema fields: The schema includes an NFR attribute, an ISO/IEC 25010-aligned intent, nested characteristic fields, imperative constraints, and checkable acceptance criteria.Core fields cover performance, error handling, code smell, and readability, with constraints scoped to the relevant quality attribute.
  • Variation design: Acceptance criteria never reference HumanEval tests, while prompt variations change lexical choice and clause order in intent and constraints.The supplied schema description distinguishes checkable requirements from benchmark-specific test references.
Loading 2608.13742v1…